2024-04-27T04:13:44.802 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-27T04:13:44.802 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests} --verbose --owner scheduled_teuthology@teuthology --name teuthology-2024-04-26_22:40:03-rgw-reef-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-26_22:40:03-rgw-reef-distro-default-smithi/7675694 -- /home/teuthworker/archive/teuthology-2024-04-26_22:40:03-rgw-reef-distro-default-smithi/7675694/orig.config.yaml 2024-04-27T04:13:44.830 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T04:13:44.910 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-26_22:40:03-rgw-reef-distro-default-smithi/7675694 branch: reef description: rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7675694' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-26_22:40:03-rgw-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: reef ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw lc debug interval: 10 setgroup: ceph setuser: ceph storage classes: LUKEWARM, FROZEN mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 7d2137f0a666c92a9411c1af5c33e4968be54da6 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 7d2137f0a666c92a9411c1af5c33e4968be54da6 rgw: frontend: beast storage classes: LUKEWARM, FROZEN s3tests: force-branch: ceph-reef selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: reef sha1: 5403a02127f27ec6c3309ef057522ab70cdeb9d2 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - mon.a - osd.0 - osd.1 - osd.2 - mgr.0 - client.0 seed: 4225 sha1: 7d2137f0a666c92a9411c1af5c33e4968be54da6 sleep_before_teardown: 0 subset: 0/1 suite: rgw suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_5403a02127f27ec6c3309ef057522ab70cdeb9d2/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 5403a02127f27ec6c3309ef057522ab70cdeb9d2 targets: smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAPGQNyqJD15fzyQVwadUa2FwDkQemrVVQxFyYqU2MnI4Jl0omGnJaswkXZsnzsgOs5w6pf+Ebgd2mmMp+O+Xec= tasks: - install: null - ceph: null - rgw: - client.0 - tox: - client.0 - s3tests: client.0: extra_attrs: - lifecycle lc_debug_interval: 10 rgw_server: client.0 storage classes: LUKEWARM, FROZEN teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_22:40:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-27T04:13:44.911 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_5403a02127f27ec6c3309ef057522ab70cdeb9d2/qa; will attempt to use it 2024-04-27T04:13:44.911 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_5403a02127f27ec6c3309ef057522ab70cdeb9d2/qa/tasks 2024-04-27T04:13:44.911 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-27T04:13:44.913 INFO:teuthology.task.internal:Checking packages... 2024-04-27T04:13:44.937 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '7d2137f0a666c92a9411c1af5c33e4968be54da6' 2024-04-27T04:13:44.937 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T04:13:44.937 INFO:teuthology.packaging:ref: None 2024-04-27T04:13:44.937 INFO:teuthology.packaging:tag: None 2024-04-27T04:13:44.937 INFO:teuthology.packaging:branch: reef 2024-04-27T04:13:44.937 INFO:teuthology.packaging:sha1: 7d2137f0a666c92a9411c1af5c33e4968be54da6 2024-04-27T04:13:44.937 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef 2024-04-27T04:13:45.206 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1260.g5403a021 2024-04-27T04:13:45.207 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-27T04:13:45.214 INFO:teuthology.task.internal:no buildpackages task found 2024-04-27T04:13:45.214 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-27T04:13:45.233 INFO:teuthology.task.internal:Saving configuration 2024-04-27T04:13:45.243 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-27T04:13:45.249 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T04:13:45.275 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_22:40:03-rgw-reef-distro-default-smithi/7675694', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 04:08:19.428290', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAPGQNyqJD15fzyQVwadUa2FwDkQemrVVQxFyYqU2MnI4Jl0omGnJaswkXZsnzsgOs5w6pf+Ebgd2mmMp+O+Xec='} 2024-04-27T04:13:45.275 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-27T04:13:45.281 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'osd.2', 'mgr.0', 'client.0'] 2024-04-27T04:13:45.281 INFO:teuthology.run_tasks:Running task console_log... 2024-04-27T04:13:45.322 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f667b2a4f70>, signals=[15]) 2024-04-27T04:13:45.323 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-27T04:13:45.329 INFO:teuthology.task.internal:Opening connections... 2024-04-27T04:13:45.329 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2024-04-27T04:13:45.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:13:45.407 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-27T04:13:45.415 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2024-04-27T04:13:45.434 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2024-04-27T04:13:45.434 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2024-04-27T04:13:45.494 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="Red Hat Enterprise Linux" 2024-04-27T04:13:45.494 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="8.6 (Ootpa)" 2024-04-27T04:13:45.494 INFO:teuthology.orchestra.run.smithi161.stdout:ID="rhel" 2024-04-27T04:13:45.494 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="fedora" 2024-04-27T04:13:45.494 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="8.6" 2024-04-27T04:13:45.494 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el8" 2024-04-27T04:13:45.495 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-27T04:13:45.495 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2024-04-27T04:13:45.495 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-27T04:13:45.495 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://www.redhat.com/" 2024-04-27T04:13:45.495 INFO:teuthology.orchestra.run.smithi161.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-27T04:13:45.495 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-27T04:13:45.495 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:13:45.495 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-27T04:13:45.495 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-27T04:13:45.495 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-27T04:13:45.495 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-27T04:13:45.496 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2024-04-27T04:13:45.523 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-27T04:13:45.531 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-27T04:13:45.589 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-27T04:13:45.589 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2024-04-27T04:13:45.608 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-27T04:13:45.615 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-27T04:13:45.615 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2024-04-27T04:13:45.670 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-27T04:13:45.703 INFO:teuthology.run_tasks:Running task kernel... 2024-04-27T04:13:45.722 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-27T04:13:45.722 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-27T04:13:45.723 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T04:13:45.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T04:13:45.741 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-04-27T04:13:45.801 INFO:teuthology.orchestra.run.smithi161.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-27T04:13:45.801 INFO:teuthology.task.kernel:Running kernel on smithi161: 4.18.0-513.9.1.el8_9.x86_64 2024-04-27T04:13:45.802 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum install -y kernel 2024-04-27T04:13:47.263 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:13:47.263 INFO:teuthology.orchestra.run.smithi161.stdout:Unable to read consumer identity 2024-04-27T04:13:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:13:47.377 INFO:teuthology.orchestra.run.smithi161.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-27T04:13:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:13:49.277 INFO:teuthology.orchestra.run.smithi161.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-04-27T04:13:50.829 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-27T04:13:54.452 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 395 kB/s | 24 kB 00:00 2024-04-27T04:13:55.179 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-27T04:13:55.180 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-27T04:13:55.256 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:13:55.257 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-04-27T04:13:55.257 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:13:55.396 DEBUG:teuthology.orchestra.run.smithi161:> echo no | sudo yum reinstall kernel || true 2024-04-27T04:13:55.732 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:13:55.732 INFO:teuthology.orchestra.run.smithi161.stdout:Unable to read consumer identity 2024-04-27T04:13:55.785 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:13:55.785 INFO:teuthology.orchestra.run.smithi161.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-27T04:13:55.785 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:13:56.105 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:00:02 ago on Sat 27 Apr 2024 04:13:54 AM UTC. 2024-04-27T04:13:56.186 INFO:teuthology.orchestra.run.smithi161.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-27T04:13:56.201 INFO:teuthology.orchestra.run.smithi161.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-27T04:13:56.240 INFO:teuthology.orchestra.run.smithi161.stderr:Error: No packages marked for reinstall. 2024-04-27T04:13:56.287 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum reinstall -y kernel || true 2024-04-27T04:13:56.621 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:13:56.621 INFO:teuthology.orchestra.run.smithi161.stdout:Unable to read consumer identity 2024-04-27T04:13:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:13:56.676 INFO:teuthology.orchestra.run.smithi161.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-27T04:13:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:13:56.982 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:00:02 ago on Sat 27 Apr 2024 04:13:54 AM UTC. 2024-04-27T04:13:57.046 INFO:teuthology.orchestra.run.smithi161.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-27T04:13:57.046 INFO:teuthology.orchestra.run.smithi161.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-27T04:13:57.076 INFO:teuthology.orchestra.run.smithi161.stderr:Error: No packages marked for reinstall. 2024-04-27T04:13:57.128 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2024-04-27T04:13:57.305 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-27T04:13:57.305 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-27T04:13:57.305 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-27T04:13:57.305 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-27T04:13:57.306 DEBUG:teuthology.orchestra.run.smithi161:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T04:13:57.339 INFO:teuthology.orchestra.run.smithi161.stdout:ttyS1 2024-04-27T04:13:57.351 DEBUG:teuthology.parallel:result is None 2024-04-27T04:13:57.351 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-27T04:13:57.358 INFO:teuthology.task.internal:Creating test directory... 2024-04-27T04:13:57.358 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T04:13:57.407 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-27T04:13:57.414 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-27T04:13:57.421 INFO:teuthology.task.internal:Creating archive directory... 2024-04-27T04:13:57.421 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T04:13:57.476 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-27T04:13:57.483 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-27T04:13:57.483 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T04:13:57.544 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T04:13:57.554 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T04:13:57.555 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-27T04:13:57.562 INFO:teuthology.task.internal:Configuring sudo... 2024-04-27T04:13:57.562 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T04:13:57.622 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-27T04:13:57.631 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-27T04:13:57.631 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T04:13:57.680 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T04:13:57.770 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T04:13:57.821 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T04:13:57.903 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T04:13:57.950 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:13:57.951 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T04:13:58.020 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2024-04-27T04:13:58.109 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T04:13:58.373 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-27T04:13:58.382 INFO:teuthology.task.internal:Starting timer... 2024-04-27T04:13:58.382 INFO:teuthology.run_tasks:Running task pcp... 2024-04-27T04:13:58.398 INFO:teuthology.run_tasks:Running task selinux... 2024-04-27T04:13:58.410 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-27T04:13:58.410 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2024-04-27T04:13:58.469 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2024-04-27T04:13:58.471 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-27T04:13:58.471 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2024-04-27T04:13:58.498 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2024-04-27T04:13:58.498 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi161.front.sepia.ceph.com': 'permissive'} 2024-04-27T04:13:58.498 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-27T04:13:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T04:13:58.567 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-27T04:13:58.567 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2024-04-27T04:13:58.635 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-27T04:13:58.644 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-27T04:13:58.651 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-27T04:13:58.780 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-27T04:13:58.797 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-27T04:13:58.798 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi161.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-27T04:21:50.498 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi161.front.sepia.ceph.com')] 2024-04-27T04:21:50.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-27T04:21:50.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T04:21:50.575 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-04-27T04:21:50.644 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-27T04:21:50.644 INFO:teuthology.run_tasks:Running task clock... 2024-04-27T04:21:50.654 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-27T04:21:50.655 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T04:21:50.655 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T04:21:50.717 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-27T04:21:50.732 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-27T04:21:50.819 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: ntpd: command not found 2024-04-27T04:21:50.831 INFO:teuthology.orchestra.run.smithi161.stdout:506 Cannot talk to daemon 2024-04-27T04:21:50.846 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-27T04:21:50.860 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-27T04:21:50.940 INFO:teuthology.orchestra.run.smithi161.stderr:bash: ntpq: command not found 2024-04-27T04:21:50.945 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T04:21:50.945 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-04-27T04:21:50.945 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T04:21:50.945 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T04:21:50.945 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T04:21:50.945 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T04:21:50.945 INFO:teuthology.run_tasks:Running task install... 2024-04-27T04:21:50.954 DEBUG:teuthology.task.install:project ceph 2024-04-27T04:21:50.954 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7d2137f0a666c92a9411c1af5c33e4968be54da6'}} 2024-04-27T04:21:50.954 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7d2137f0a666c92a9411c1af5c33e4968be54da6'} 2024-04-27T04:21:50.954 INFO:teuthology.task.install:Using flavor: default 2024-04-27T04:21:50.960 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-27T04:21:50.960 INFO:teuthology.task.install:extra packages: [] 2024-04-27T04:21:50.961 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7d2137f0a666c92a9411c1af5c33e4968be54da6', 'tag': None, 'wait_for_package': False} 2024-04-27T04:21:50.961 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7d2137f0a666c92a9411c1af5c33e4968be54da6 2024-04-27T04:21:51.187 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/reef/7d2137f0a666c92a9411c1af5c33e4968be54da6/centos/8/flavors/default/ 2024-04-27T04:21:51.188 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1258.g7d2137f0 2024-04-27T04:21:51.301 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/reef/7d2137f0a666c92a9411c1af5c33e4968be54da6/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/7d2137f0a666c92a9411c1af5c33e4968be54da6/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/7d2137f0a666c92a9411c1af5c33e4968be54da6/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-27T04:21:51.302 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:21:51.302 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-27T04:21:51.330 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-27T04:21:51.331 DEBUG:teuthology.orchestra.run.smithi161:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7d2137f0a666c92a9411c1af5c33e4968be54da6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-27T04:21:51.399 DEBUG:teuthology.orchestra.run.smithi161:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-27T04:21:51.476 DEBUG:teuthology.orchestra.run.smithi161:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-27T04:21:51.543 INFO:teuthology.orchestra.run.smithi161.stdout:check_obsoletes = 1 2024-04-27T04:21:51.544 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum clean all 2024-04-27T04:21:51.921 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:21:52.361 INFO:teuthology.orchestra.run.smithi161.stdout:63 files removed 2024-04-27T04:21:52.414 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw 2024-04-27T04:21:52.748 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:21:53.733 INFO:teuthology.orchestra.run.smithi161.stdout:ceph packages for x86_64 174 kB/s | 80 kB 00:00 2024-04-27T04:21:54.334 INFO:teuthology.orchestra.run.smithi161.stdout:ceph noarch packages 20 kB/s | 11 kB 00:00 2024-04-27T04:21:54.767 INFO:teuthology.orchestra.run.smithi161.stdout:ceph source packages 3.7 kB/s | 1.5 kB 00:00 2024-04-27T04:21:55.010 INFO:teuthology.orchestra.run.smithi161.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-04-27T04:22:08.626 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 1.2 MB/s | 16 MB 00:13 2024-04-27T04:22:12.242 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 388 kB/s | 24 kB 00:00 2024-04-27T04:22:13.411 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 MB/s | 67 MB 00:01 2024-04-27T04:22:23.920 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 62 MB/s | 60 MB 00:00 2024-04-27T04:22:34.478 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 24 MB/s | 8.8 MB 00:00 2024-04-27T04:22:41.677 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:22:41.679 INFO:teuthology.orchestra.run.smithi161.stdout:==================================================================================================== 2024-04-27T04:22:41.679 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-04-27T04:22:41.679 INFO:teuthology.orchestra.run.smithi161.stdout:==================================================================================================== 2024-04-27T04:22:41.679 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:22:41.679 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 13 M 2024-04-27T04:22:41.679 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2024-04-27T04:22:41.679 INFO:teuthology.orchestra.run.smithi161.stdout: librados2 x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 3.7 M 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1 x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 3.6 M 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 5.5 M 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 22 M 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 24 k 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2 x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 780 k 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1 x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 473 k 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2 x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 6.5 M 2024-04-27T04:22:41.680 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 45 k 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 129 k 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 197 k 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 370 k 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 364 k 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 107 k 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout:==================================================================================================== 2024-04-27T04:22:41.681 INFO:teuthology.orchestra.run.smithi161.stdout:Install 21 Packages 2024-04-27T04:22:41.682 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 2 Packages 2024-04-27T04:22:41.682 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:22:41.682 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 59 M 2024-04-27T04:22:41.682 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:22:42.337 INFO:teuthology.orchestra.run.smithi161.stdout:(1/23): ceph-base-18.2.2-1258.g7d2137f0.el8.x86 11 MB/s | 5.5 MB 00:00 2024-04-27T04:22:42.404 INFO:teuthology.orchestra.run.smithi161.stdout:(2/23): ceph-selinux-18.2.2-1258.g7d2137f0.el8. 364 kB/s | 24 kB 00:00 2024-04-27T04:22:42.479 INFO:teuthology.orchestra.run.smithi161.stdout:(3/23): libcephfs2-18.2.2-1258.g7d2137f0.el8.x8 10 MB/s | 780 kB 00:00 2024-04-27T04:22:42.538 INFO:teuthology.orchestra.run.smithi161.stdout:(4/23): libradosstriper1-18.2.2-1258.g7d2137f0. 7.9 MB/s | 473 kB 00:00 2024-04-27T04:22:42.905 INFO:teuthology.orchestra.run.smithi161.stdout:(5/23): librgw2-18.2.2-1258.g7d2137f0.el8.x86_6 18 MB/s | 6.5 MB 00:00 2024-04-27T04:22:42.955 INFO:teuthology.orchestra.run.smithi161.stdout:(6/23): python3-ceph-argparse-18.2.2-1258.g7d21 898 kB/s | 45 kB 00:00 2024-04-27T04:22:43.005 INFO:teuthology.orchestra.run.smithi161.stdout:(7/23): python3-ceph-common-18.2.2-1258.g7d2137 2.5 MB/s | 129 kB 00:00 2024-04-27T04:22:43.064 INFO:teuthology.orchestra.run.smithi161.stdout:(8/23): python3-cephfs-18.2.2-1258.g7d2137f0.el 3.3 MB/s | 197 kB 00:00 2024-04-27T04:22:43.130 INFO:teuthology.orchestra.run.smithi161.stdout:(9/23): python3-rados-18.2.2-1258.g7d2137f0.el8 5.4 MB/s | 370 kB 00:00 2024-04-27T04:22:43.189 INFO:teuthology.orchestra.run.smithi161.stdout:(10/23): python3-rbd-18.2.2-1258.g7d2137f0.el8. 6.1 MB/s | 364 kB 00:00 2024-04-27T04:22:43.248 INFO:teuthology.orchestra.run.smithi161.stdout:(11/23): python3-rgw-18.2.2-1258.g7d2137f0.el8. 1.8 MB/s | 107 kB 00:00 2024-04-27T04:22:43.415 INFO:teuthology.orchestra.run.smithi161.stdout:(12/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2024-04-27T04:22:43.457 INFO:teuthology.orchestra.run.smithi161.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2024-04-27T04:22:43.499 INFO:teuthology.orchestra.run.smithi161.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2024-04-27T04:22:43.649 INFO:teuthology.orchestra.run.smithi161.stdout:(15/23): thrift-0.13.0-2.el8.x86_64.rpm 12 MB/s | 1.7 MB 00:00 2024-04-27T04:22:43.741 INFO:teuthology.orchestra.run.smithi161.stdout:(16/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 515 kB/s | 47 kB 00:00 2024-04-27T04:22:43.800 INFO:teuthology.orchestra.run.smithi161.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 755 kB/s | 44 kB 00:00 2024-04-27T04:22:43.858 INFO:teuthology.orchestra.run.smithi161.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.4 MB/s | 259 kB 00:00 2024-04-27T04:22:43.948 INFO:teuthology.orchestra.run.smithi161.stdout:(19/23): librdkafka-0.11.4-3.el8.x86_64.rpm 3.8 MB/s | 354 kB 00:00 2024-04-27T04:22:44.392 INFO:teuthology.orchestra.run.smithi161.stdout:(20/23): librados2-18.2.2-1258.g7d2137f0.el8.x8 8.3 MB/s | 3.7 MB 00:00 2024-04-27T04:22:44.779 INFO:teuthology.orchestra.run.smithi161.stdout:(21/23): librbd1-18.2.2-1258.g7d2137f0.el8.x86_ 9.3 MB/s | 3.6 MB 00:00 2024-04-27T04:22:45.009 INFO:teuthology.orchestra.run.smithi161.stdout:(22/23): ceph-common-18.2.2-1258.g7d2137f0.el8. 6.9 MB/s | 22 MB 00:03 2024-04-27T04:22:45.185 INFO:teuthology.orchestra.run.smithi161.stdout:(23/23): ceph-radosgw-18.2.2-1258.g7d2137f0.el8 3.9 MB/s | 13 MB 00:03 2024-04-27T04:22:45.188 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:22:45.188 INFO:teuthology.orchestra.run.smithi161.stdout:Total 17 MB/s | 59 MB 00:03 2024-04-27T04:22:45.238 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:22:45.284 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:22:45.285 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:22:45.622 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:22:45.631 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:22:47.010 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:22:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-27T04:22:47.376 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-27T04:22:47.785 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-27T04:22:48.026 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librados2-2:18.2.2-1258.g7d2137f0.el8.x86_64 3/25 2024-04-27T04:22:48.125 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:18.2.2-1258.g7d2137f0.el8.x86_64 3/25 2024-04-27T04:22:48.281 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-2:18.2.2-1258.g7d2137f0.el8.x86_64 4/25 2024-04-27T04:22:48.532 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-27T04:22:48.751 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-2:18.2.2-1258.g7d2137f0.el8.x86_64 6/25 2024-04-27T04:22:49.139 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-2:18.2.2-1258.g7d2137f0.el8.x86_64 6/25 2024-04-27T04:22:49.334 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librbd1-2:18.2.2-1258.g7d2137f0.el8.x86_64 7/25 2024-04-27T04:22:49.425 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:18.2.2-1258.g7d2137f0.el8.x86_64 7/25 2024-04-27T04:22:49.618 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-27T04:22:49.682 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-27T04:22:49.841 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 9/25 2024-04-27T04:22:50.004 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-argparse-2:18.2.2-1258.g7d2137f0.el8. 10/25 2024-04-27T04:22:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-2:18.2.2-1258.g7d2137f0.el8.x86_64 11/25 2024-04-27T04:22:50.468 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-2:18.2.2-1258.g7d2137f0.el8.x86_64 12/25 2024-04-27T04:22:50.617 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradosstriper1-2:18.2.2-1258.g7d2137f0.el8.x86_6 13/25 2024-04-27T04:22:50.689 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1258.g7d2137f0.el8.x86_6 13/25 2024-04-27T04:22:50.859 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-27T04:22:51.042 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-27T04:22:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-27T04:22:52.125 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librgw2-2:18.2.2-1258.g7d2137f0.el8.x86_64 17/25 2024-04-27T04:22:52.203 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librgw2-2:18.2.2-1258.g7d2137f0.el8.x86_64 17/25 2024-04-27T04:22:52.460 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-2:18.2.2-1258.g7d2137f0.el8.x86_64 18/25 2024-04-27T04:22:52.618 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-common-2:18.2.2-1258.g7d2137f0.el8.x8 19/25 2024-04-27T04:22:56.381 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:18.2.2-1258.g7d2137f0.el8.x86_64 20/25 2024-04-27T04:22:56.859 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-2:18.2.2-1258.g7d2137f0.el8.x86_64 20/25 2024-04-27T04:22:57.456 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:18.2.2-1258.g7d2137f0.el8.x86_64 20/25 2024-04-27T04:22:57.732 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-2:18.2.2-1258.g7d2137f0.el8.x86_64 21/25 2024-04-27T04:22:57.797 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-2:18.2.2-1258.g7d2137f0.el8.x86_64 21/25 2024-04-27T04:22:57.958 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-selinux-2:18.2.2-1258.g7d2137f0.el8.x86_64 22/25 2024-04-27T04:23:17.418 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1258.g7d2137f0.el8.x86_64 22/25 2024-04-27T04:23:17.419 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /sys 2024-04-27T04:23:17.419 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /proc 2024-04-27T04:23:17.419 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /dev 2024-04-27T04:23:17.419 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /run 2024-04-27T04:23:17.419 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /mnt 2024-04-27T04:23:17.419 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /var/tmp 2024-04-27T04:23:17.419 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /home 2024-04-27T04:23:17.419 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /tmp 2024-04-27T04:23:17.419 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /dev 2024-04-27T04:23:17.419 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:23:18.770 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-2:18.2.2-1258.g7d2137f0.el8.x86_64 23/25 2024-04-27T04:23:18.792 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1258.g7d2137f0.el8.x86_64 23/25 2024-04-27T04:23:18.792 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-27T04:23:19.040 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-27T04:23:19.041 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-27T04:23:19.725 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-27T04:23:19.725 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/25 2024-04-27T04:23:19.726 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-2:18.2.2-1258.g7d2137f0.el8.x86_64 2/25 2024-04-27T04:23:19.726 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-2:18.2.2-1258.g7d2137f0.el8.x86_64 3/25 2024-04-27T04:23:19.726 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-selinux-2:18.2.2-1258.g7d2137f0.el8.x86_64 4/25 2024-04-27T04:23:19.726 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-2:18.2.2-1258.g7d2137f0.el8.x86_64 5/25 2024-04-27T04:23:19.726 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradosstriper1-2:18.2.2-1258.g7d2137f0.el8.x86_6 6/25 2024-04-27T04:23:19.727 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librgw2-2:18.2.2-1258.g7d2137f0.el8.x86_64 7/25 2024-04-27T04:23:19.727 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1258.g7d2137f0.el8. 8/25 2024-04-27T04:23:19.727 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-common-2:18.2.2-1258.g7d2137f0.el8.x8 9/25 2024-04-27T04:23:19.727 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-2:18.2.2-1258.g7d2137f0.el8.x86_64 10/25 2024-04-27T04:23:19.727 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-2:18.2.2-1258.g7d2137f0.el8.x86_64 11/25 2024-04-27T04:23:19.727 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-2:18.2.2-1258.g7d2137f0.el8.x86_64 12/25 2024-04-27T04:23:19.728 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-2:18.2.2-1258.g7d2137f0.el8.x86_64 13/25 2024-04-27T04:23:19.728 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-27T04:23:19.728 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-27T04:23:19.728 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-27T04:23:19.728 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-27T04:23:19.728 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 18/25 2024-04-27T04:23:19.728 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2024-04-27T04:23:19.728 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2024-04-27T04:23:19.728 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2024-04-27T04:23:19.728 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:18.2.2-1258.g7d2137f0.el8.x86_64 22/25 2024-04-27T04:23:19.728 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-27T04:23:19.728 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:18.2.2-1258.g7d2137f0.el8.x86_64 24/25 2024-04-27T04:23:24.626 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-27T04:23:24.627 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:23:25.201 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:23:25.201 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2024-04-27T04:23:25.201 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.201 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.201 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:23:25.201 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.201 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.201 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.201 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-27T04:23:25.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:25.203 INFO:teuthology.orchestra.run.smithi161.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-27T04:23:25.203 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:23:25.203 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:23:25.397 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-test 2024-04-27T04:23:25.753 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:23:26.482 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 138 kB/s | 4.1 kB 00:00 2024-04-27T04:23:26.734 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 142 kB/s | 4.5 kB 00:00 2024-04-27T04:23:29.333 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:23:29.334 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-04-27T04:23:29.334 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-04-27T04:23:29.334 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-04-27T04:23:29.334 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:23:29.334 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 47 M 2024-04-27T04:23:29.334 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-04-27T04:23:29.334 INFO:teuthology.orchestra.run.smithi161.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-27T04:23:29.334 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 142 k 2024-04-27T04:23:29.334 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-27T04:23:29.334 INFO:teuthology.orchestra.run.smithi161.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-27T04:23:29.334 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-27T04:23:29.335 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:23:29.335 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:23:29.335 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-04-27T04:23:29.335 INFO:teuthology.orchestra.run.smithi161.stdout:Install 6 Packages 2024-04-27T04:23:29.335 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:23:29.335 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 48 M 2024-04-27T04:23:29.336 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 213 M 2024-04-27T04:23:29.336 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:23:29.402 INFO:teuthology.orchestra.run.smithi161.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 70 kB 00:00 2024-04-27T04:23:29.502 INFO:teuthology.orchestra.run.smithi161.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.3 MB/s | 323 kB 00:00 2024-04-27T04:23:29.552 INFO:teuthology.orchestra.run.smithi161.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 4.0 MB/s | 202 kB 00:00 2024-04-27T04:23:29.611 INFO:teuthology.orchestra.run.smithi161.stdout:(4/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-04-27T04:23:29.778 INFO:teuthology.orchestra.run.smithi161.stdout:(5/6): libcephsqlite-18.2.2-1258.g7d2137f0.el8. 321 kB/s | 142 kB 00:00 2024-04-27T04:23:31.703 INFO:teuthology.orchestra.run.smithi161.stdout:(6/6): ceph-test-18.2.2-1258.g7d2137f0.el8.x86_ 20 MB/s | 47 MB 00:02 2024-04-27T04:23:31.705 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:23:31.706 INFO:teuthology.orchestra.run.smithi161.stdout:Total 20 MB/s | 48 MB 00:02 2024-04-27T04:23:31.744 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:23:31.766 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:23:31.766 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:23:32.041 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:23:32.050 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:23:32.525 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:23:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-27T04:23:32.765 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-27T04:23:32.989 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-27T04:23:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-27T04:23:33.357 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-27T04:23:33.519 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephsqlite-2:18.2.2-1258.g7d2137f0.el8.x86_64 5/6 2024-04-27T04:23:38.026 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1258.g7d2137f0.el8.x86_64 5/6 2024-04-27T04:23:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-2:18.2.2-1258.g7d2137f0.el8.x86_64 6/6 2024-04-27T04:23:38.544 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-test-2:18.2.2-1258.g7d2137f0.el8.x86_64 6/6 2024-04-27T04:23:38.544 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/6 2024-04-27T04:23:38.544 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephsqlite-2:18.2.2-1258.g7d2137f0.el8.x86_64 2/6 2024-04-27T04:23:38.544 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-27T04:23:38.544 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-27T04:23:38.544 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-27T04:23:43.167 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-27T04:23:43.167 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:23:43.709 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:23:43.709 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:23:43.709 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:43.709 INFO:teuthology.orchestra.run.smithi161.stdout: jq-1.6-7.el8.x86_64 2024-04-27T04:23:43.709 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:23:43.709 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-27T04:23:43.709 INFO:teuthology.orchestra.run.smithi161.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-27T04:23:43.709 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-27T04:23:43.709 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:23:43.709 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:23:43.877 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph 2024-04-27T04:23:44.217 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:23:44.975 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 138 kB/s | 4.1 kB 00:00 2024-04-27T04:23:45.236 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 170 kB/s | 4.5 kB 00:00 2024-04-27T04:23:47.837 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:23:47.841 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================= 2024-04-27T04:23:47.841 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-04-27T04:23:47.841 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================= 2024-04-27T04:23:47.841 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:23:47.841 INFO:teuthology.orchestra.run.smithi161.stdout: ceph x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 6.4 k 2024-04-27T04:23:47.841 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-04-27T04:23:47.841 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 2.3 M 2024-04-27T04:23:47.841 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 1.6 M 2024-04-27T04:23:47.841 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1258.g7d2137f0.el8 ceph-noarch 245 k 2024-04-27T04:23:47.841 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 4.8 M 2024-04-27T04:23:47.841 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 18 M 2024-04-27T04:23:47.841 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-27T04:23:47.842 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:23:47.843 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:23:47.844 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================= 2024-04-27T04:23:47.844 INFO:teuthology.orchestra.run.smithi161.stdout:Install 32 Packages 2024-04-27T04:23:47.844 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:23:47.844 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 31 M 2024-04-27T04:23:47.844 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 124 M 2024-04-27T04:23:47.844 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:23:48.112 INFO:teuthology.orchestra.run.smithi161.stdout:(1/32): ceph-18.2.2-1258.g7d2137f0.el8.x86_64.r 49 kB/s | 6.4 kB 00:00 2024-04-27T04:23:48.262 INFO:teuthology.orchestra.run.smithi161.stdout:(2/32): ceph-mgr-18.2.2-1258.g7d2137f0.el8.x86_ 5.6 MB/s | 1.6 MB 00:00 2024-04-27T04:23:48.863 INFO:teuthology.orchestra.run.smithi161.stdout:(3/32): ceph-mds-18.2.2-1258.g7d2137f0.el8.x86_ 2.6 MB/s | 2.3 MB 00:00 2024-04-27T04:23:49.263 INFO:teuthology.orchestra.run.smithi161.stdout:(4/32): ceph-osd-18.2.2-1258.g7d2137f0.el8.x86_ 18 MB/s | 18 MB 00:01 2024-04-27T04:23:49.297 INFO:teuthology.orchestra.run.smithi161.stdout:(5/32): ceph-mgr-modules-core-18.2.2-1258.g7d21 566 kB/s | 245 kB 00:00 2024-04-27T04:23:49.405 INFO:teuthology.orchestra.run.smithi161.stdout:(6/32): ceph-mon-18.2.2-1258.g7d2137f0.el8.x86_ 3.7 MB/s | 4.8 MB 00:01 2024-04-27T04:23:49.572 INFO:teuthology.orchestra.run.smithi161.stdout:(7/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 141 kB/s | 44 kB 00:00 2024-04-27T04:23:49.714 INFO:teuthology.orchestra.run.smithi161.stdout:(8/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 444 kB/s | 185 kB 00:00 2024-04-27T04:23:49.756 INFO:teuthology.orchestra.run.smithi161.stdout:(9/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 499 kB/s | 173 kB 00:00 2024-04-27T04:23:49.790 INFO:teuthology.orchestra.run.smithi161.stdout:(10/32): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2024-04-27T04:23:49.839 INFO:teuthology.orchestra.run.smithi161.stdout:(11/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-04-27T04:23:49.865 INFO:teuthology.orchestra.run.smithi161.stdout:(12/32): python3-jaraco-functools-2.0-4.el8.noa 161 kB/s | 18 kB 00:00 2024-04-27T04:23:49.890 INFO:teuthology.orchestra.run.smithi161.stdout:(13/32): python3-logutils-0.3.5-11.el8.noarch.r 485 kB/s | 49 kB 00:00 2024-04-27T04:23:49.924 INFO:teuthology.orchestra.run.smithi161.stdout:(14/32): python3-more-itertools-7.2.0-3.el8.noa 689 kB/s | 59 kB 00:00 2024-04-27T04:23:49.974 INFO:teuthology.orchestra.run.smithi161.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2024-04-27T04:23:50.000 INFO:teuthology.orchestra.run.smithi161.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 249 kB/s | 19 kB 00:00 2024-04-27T04:23:50.025 INFO:teuthology.orchestra.run.smithi161.stdout:(17/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2024-04-27T04:23:50.050 INFO:teuthology.orchestra.run.smithi161.stdout:(18/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2024-04-27T04:23:50.075 INFO:teuthology.orchestra.run.smithi161.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2024-04-27T04:23:50.101 INFO:teuthology.orchestra.run.smithi161.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 366 kB/s | 27 kB 00:00 2024-04-27T04:23:50.168 INFO:teuthology.orchestra.run.smithi161.stdout:(21/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.1 MB/s | 255 kB 00:00 2024-04-27T04:23:50.193 INFO:teuthology.orchestra.run.smithi161.stdout:(22/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2024-04-27T04:23:50.243 INFO:teuthology.orchestra.run.smithi161.stdout:(23/32): python3-webtest-2.0.33-1.el8.noarch.rp 600 kB/s | 85 kB 00:00 2024-04-27T04:23:50.268 INFO:teuthology.orchestra.run.smithi161.stdout:(24/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2024-04-27T04:23:50.327 INFO:teuthology.orchestra.run.smithi161.stdout:(25/32): python3-pycparser-2.14-14.el8.noarch.r 816 kB/s | 109 kB 00:00 2024-04-27T04:23:50.352 INFO:teuthology.orchestra.run.smithi161.stdout:(26/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.1 MB/s | 238 kB 00:00 2024-04-27T04:23:50.386 INFO:teuthology.orchestra.run.smithi161.stdout:(27/32): python3-cryptography-3.2.1-7.el8_9.x86 5.1 MB/s | 559 kB 00:00 2024-04-27T04:23:50.411 INFO:teuthology.orchestra.run.smithi161.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 665 kB/s | 39 kB 00:00 2024-04-27T04:23:50.436 INFO:teuthology.orchestra.run.smithi161.stdout:(29/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 2.0 MB/s | 103 kB 00:00 2024-04-27T04:23:50.462 INFO:teuthology.orchestra.run.smithi161.stdout:(30/32): python3-mako-1.0.6-14.el8.noarch.rpm 3.0 MB/s | 157 kB 00:00 2024-04-27T04:23:50.495 INFO:teuthology.orchestra.run.smithi161.stdout:(31/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.7 MB/s | 457 kB 00:00 2024-04-27T04:23:50.520 INFO:teuthology.orchestra.run.smithi161.stdout:(32/32): python3-pytz-2017.2-11.el8.noarch.rpm 650 kB/s | 54 kB 00:00 2024-04-27T04:23:50.525 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:23:50.525 INFO:teuthology.orchestra.run.smithi161.stdout:Total 12 MB/s | 31 MB 00:02 2024-04-27T04:23:50.620 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:23:50.658 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:23:50.658 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:23:50.883 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:23:50.891 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:23:52.055 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:23:53.011 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-27T04:23:53.240 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-27T04:23:53.396 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 3/32 2024-04-27T04:23:53.624 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2024-04-27T04:23:53.945 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 5/32 2024-04-27T04:23:54.180 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/32 2024-04-27T04:23:54.403 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2024-04-27T04:23:54.759 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 8/32 2024-04-27T04:23:55.044 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 9/32 2024-04-27T04:23:55.240 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2024-04-27T04:23:55.425 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2024-04-27T04:23:55.696 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2024-04-27T04:23:55.957 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-27T04:23:56.172 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-27T04:23:56.370 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-27T04:23:56.567 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-27T04:23:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-27T04:23:56.981 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-27T04:23:57.211 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-27T04:23:57.402 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-27T04:23:57.613 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-27T04:23:57.853 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-27T04:23:58.096 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-27T04:23:58.358 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-27T04:23:58.662 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-27T04:23:58.989 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-27T04:23:59.377 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1258.g7d2137f0.el8. 27/32 2024-04-27T04:23:59.539 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-2:18.2.2-1258.g7d2137f0.el8.x86_64 28/32 2024-04-27T04:24:01.181 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1258.g7d2137f0.el8.x86_64 28/32 2024-04-27T04:24:01.389 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-2:18.2.2-1258.g7d2137f0.el8.x86_64 29/32 2024-04-27T04:24:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-2:18.2.2-1258.g7d2137f0.el8.x86_64 29/32 2024-04-27T04:24:02.044 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-2:18.2.2-1258.g7d2137f0.el8.x86_64 30/32 2024-04-27T04:24:02.259 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-2:18.2.2-1258.g7d2137f0.el8.x86_64 30/32 2024-04-27T04:24:02.424 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-2:18.2.2-1258.g7d2137f0.el8.x86_64 31/32 2024-04-27T04:24:02.442 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-2:18.2.2-1258.g7d2137f0.el8.x86_64 31/32 2024-04-27T04:24:02.564 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-2:18.2.2-1258.g7d2137f0.el8.x86_64 32/32 2024-04-27T04:24:02.952 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-2:18.2.2-1258.g7d2137f0.el8.x86_64 32/32 2024-04-27T04:24:02.952 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/32 2024-04-27T04:24:02.952 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-2:18.2.2-1258.g7d2137f0.el8.x86_64 2/32 2024-04-27T04:24:02.952 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-2:18.2.2-1258.g7d2137f0.el8.x86_64 3/32 2024-04-27T04:24:02.952 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-2:18.2.2-1258.g7d2137f0.el8.x86_64 4/32 2024-04-27T04:24:02.952 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-2:18.2.2-1258.g7d2137f0.el8.x86_64 5/32 2024-04-27T04:24:02.952 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1258.g7d2137f0.el8. 6/32 2024-04-27T04:24:02.952 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-27T04:24:02.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-27T04:24:02.954 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-27T04:24:02.954 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-27T04:24:02.954 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-27T04:24:02.954 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2024-04-27T04:24:02.955 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 26/32 2024-04-27T04:24:02.955 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 27/32 2024-04-27T04:24:02.955 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2024-04-27T04:24:02.955 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2024-04-27T04:24:02.955 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 30/32 2024-04-27T04:24:02.955 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 31/32 2024-04-27T04:24:08.394 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 32/32 2024-04-27T04:24:08.395 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:24:09.543 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:09.543 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:24:09.543 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:24:09.543 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:24:09.543 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:24:09.543 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core-2:18.2.2-1258.g7d2137f0.el8.noarch 2024-04-27T04:24:09.543 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:24:09.543 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:24:09.543 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-27T04:24:09.543 INFO:teuthology.orchestra.run.smithi161.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-27T04:24:09.543 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend-2.6-1.el8.noarch 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-27T04:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-27T04:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-27T04:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-27T04:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-27T04:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-27T04:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-27T04:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-27T04:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-27T04:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-27T04:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-27T04:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:24:10.586 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-base 2024-04-27T04:24:10.938 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:24:11.861 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 kB/s | 4.1 kB 00:00 2024-04-27T04:24:12.128 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 121 kB/s | 4.5 kB 00:00 2024-04-27T04:24:14.547 INFO:teuthology.orchestra.run.smithi161.stdout:Package ceph-base-2:18.2.2-1258.g7d2137f0.el8.x86_64 is already installed. 2024-04-27T04:24:14.729 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:24:14.730 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-04-27T04:24:14.730 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:24:14.872 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install cephadm 2024-04-27T04:24:15.220 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:24:16.070 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 kB/s | 4.1 kB 00:00 2024-04-27T04:24:16.337 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 122 kB/s | 4.5 kB 00:00 2024-04-27T04:24:19.049 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:24:19.050 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:24:19.050 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-04-27T04:24:19.050 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:24:19.050 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:24:19.050 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm noarch 2:18.2.2-1258.g7d2137f0.el8 ceph-noarch 170 k 2024-04-27T04:24:19.050 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:19.050 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:24:19.050 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:24:19.050 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-04-27T04:24:19.050 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:19.050 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 170 k 2024-04-27T04:24:19.051 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 692 k 2024-04-27T04:24:19.051 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:24:19.474 INFO:teuthology.orchestra.run.smithi161.stdout:cephadm-18.2.2-1258.g7d2137f0.el8.noarch.rpm 402 kB/s | 170 kB 00:00 2024-04-27T04:24:19.475 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:24:19.475 INFO:teuthology.orchestra.run.smithi161.stdout:Total 400 kB/s | 170 kB 00:00 2024-04-27T04:24:19.475 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:24:19.483 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:24:19.483 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:24:19.485 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:24:19.494 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:24:19.536 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:24:21.365 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:18.2.2-1258.g7d2137f0.el8.noarch 1/1 2024-04-27T04:24:21.531 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephadm-2:18.2.2-1258.g7d2137f0.el8.noarch 1/1 2024-04-27T04:24:21.733 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:18.2.2-1258.g7d2137f0.el8.noarch 1/1 2024-04-27T04:24:26.483 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephadm-2:18.2.2-1258.g7d2137f0.el8.noarch 1/1 2024-04-27T04:24:26.484 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:24:27.067 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:27.067 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:24:27.067 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm-2:18.2.2-1258.g7d2137f0.el8.noarch 2024-04-27T04:24:27.067 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:27.068 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:24:27.257 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-immutable-object-cache 2024-04-27T04:24:27.622 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:24:28.374 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 121 kB/s | 4.1 kB 00:00 2024-04-27T04:24:28.632 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 135 kB/s | 4.5 kB 00:00 2024-04-27T04:24:31.349 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:24:31.349 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:24:31.349 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-04-27T04:24:31.349 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:24:31.349 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:24:31.349 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 165 k 2024-04-27T04:24:31.350 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:31.350 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:24:31.350 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:24:31.350 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-04-27T04:24:31.350 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:31.351 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 165 k 2024-04-27T04:24:31.351 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 503 k 2024-04-27T04:24:31.351 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:24:31.747 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-immutable-object-cache-18.2.2-1258.g7d2137 417 kB/s | 165 kB 00:00 2024-04-27T04:24:31.748 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:24:31.748 INFO:teuthology.orchestra.run.smithi161.stdout:Total 416 kB/s | 165 kB 00:00 2024-04-27T04:24:31.748 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:24:31.761 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:24:31.761 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:24:31.816 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:24:31.824 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:24:32.568 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:24:32.752 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1258.g7d2137f0. 1/1 2024-04-27T04:24:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1258.g7d2137f0. 1/1 2024-04-27T04:24:37.950 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1258.g7d2137f0. 1/1 2024-04-27T04:24:37.950 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:24:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:38.536 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:24:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:24:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:38.536 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:24:38.725 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr 2024-04-27T04:24:39.080 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:24:39.829 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 158 kB/s | 4.1 kB 00:00 2024-04-27T04:24:40.073 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 168 kB/s | 4.5 kB 00:00 2024-04-27T04:24:42.490 INFO:teuthology.orchestra.run.smithi161.stdout:Package ceph-mgr-2:18.2.2-1258.g7d2137f0.el8.x86_64 is already installed. 2024-04-27T04:24:42.679 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:24:42.680 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-04-27T04:24:42.680 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:24:42.823 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-dashboard 2024-04-27T04:24:43.170 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:24:43.929 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 121 kB/s | 4.1 kB 00:00 2024-04-27T04:24:44.182 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 126 kB/s | 4.5 kB 00:00 2024-04-27T04:24:46.785 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:24:46.786 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:24:46.786 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-04-27T04:24:46.786 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:24:46.787 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:24:46.787 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1258.g7d2137f0.el8 ceph-noarch 3.5 M 2024-04-27T04:24:46.787 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-04-27T04:24:46.787 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1258.g7d2137f0.el8 ceph-noarch 24 k 2024-04-27T04:24:46.787 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1258.g7d2137f0.el8 ceph-noarch 15 k 2024-04-27T04:24:46.787 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-27T04:24:46.787 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-27T04:24:46.787 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:46.787 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:24:46.787 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:24:46.787 INFO:teuthology.orchestra.run.smithi161.stdout:Install 5 Packages 2024-04-27T04:24:46.787 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:46.788 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 3.8 M 2024-04-27T04:24:46.788 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 87 M 2024-04-27T04:24:46.788 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:24:47.196 INFO:teuthology.orchestra.run.smithi161.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1258.g7d21 44 kB/s | 15 kB 00:00 2024-04-27T04:24:47.246 INFO:teuthology.orchestra.run.smithi161.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1258.g7d2 61 kB/s | 24 kB 00:00 2024-04-27T04:24:47.438 INFO:teuthology.orchestra.run.smithi161.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 139 kB/s | 34 kB 00:00 2024-04-27T04:24:47.630 INFO:teuthology.orchestra.run.smithi161.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 512 kB/s | 196 kB 00:00 2024-04-27T04:24:49.138 INFO:teuthology.orchestra.run.smithi161.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1258.g7d2137f0 1.5 MB/s | 3.5 MB 00:02 2024-04-27T04:24:49.140 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:24:49.141 INFO:teuthology.orchestra.run.smithi161.stdout:Total 1.6 MB/s | 3.8 MB 00:02 2024-04-27T04:24:49.141 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:24:49.150 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:24:49.150 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:24:49.223 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:24:49.232 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:24:49.488 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:24:49.706 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-27T04:24:49.919 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-27T04:24:50.048 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1258.g7d2137f0.el8.n 3/5 2024-04-27T04:24:51.233 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1258.g7d2137f0.el8. 4/5 2024-04-27T04:24:51.506 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1258.g7d2137f0.el8.noarc 5/5 2024-04-27T04:24:51.770 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1258.g7d2137f0.el8.noarc 5/5 2024-04-27T04:24:51.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1258.g7d2137f0.el8. 1/5 2024-04-27T04:24:51.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1258.g7d2137f0.el8.noarc 2/5 2024-04-27T04:24:51.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1258.g7d2137f0.el8.n 3/5 2024-04-27T04:24:51.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-27T04:24:56.445 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-27T04:24:56.445 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:24:57.017 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:57.017 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:24:57.017 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards-2:18.2.2-1258.g7d2137f0.el8.noarch 2024-04-27T04:24:57.017 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard-2:18.2.2-1258.g7d2137f0.el8.noarch 2024-04-27T04:24:57.017 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts-2:18.2.2-1258.g7d2137f0.el8.noarch 2024-04-27T04:24:57.017 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-27T04:24:57.017 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-27T04:24:57.017 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:24:57.018 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:24:57.220 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-27T04:24:57.572 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:24:58.352 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 138 kB/s | 4.1 kB 00:00 2024-04-27T04:24:58.627 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 156 kB/s | 4.5 kB 00:00 2024-04-27T04:25:01.259 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:25:01.261 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================================= 2024-04-27T04:25:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-04-27T04:25:01.261 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================================= 2024-04-27T04:25:01.261 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:25:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1258.g7d2137f0.el8 ceph-noarch 7.4 M 2024-04-27T04:25:01.261 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-04-27T04:25:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-27T04:25:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-27T04:25:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-27T04:25:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-27T04:25:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-27T04:25:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-27T04:25:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-27T04:25:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:01.262 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:25:01.262 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================================= 2024-04-27T04:25:01.262 INFO:teuthology.orchestra.run.smithi161.stdout:Install 8 Packages 2024-04-27T04:25:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:01.263 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 36 M 2024-04-27T04:25:01.263 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 207 M 2024-04-27T04:25:01.263 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:25:01.339 INFO:teuthology.orchestra.run.smithi161.stdout:(1/8): libquadmath-8.5.0-20.el8.x86_64.rpm 2.2 MB/s | 172 kB 00:00 2024-04-27T04:25:01.372 INFO:teuthology.orchestra.run.smithi161.stdout:(2/8): libgfortran-8.5.0-20.el8.x86_64.rpm 5.8 MB/s | 645 kB 00:00 2024-04-27T04:25:01.422 INFO:teuthology.orchestra.run.smithi161.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 4.4 MB/s | 225 kB 00:00 2024-04-27T04:25:01.489 INFO:teuthology.orchestra.run.smithi161.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2024-04-27T04:25:01.631 INFO:teuthology.orchestra.run.smithi161.stdout:(5/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 25 MB/s | 5.2 MB 00:00 2024-04-27T04:25:01.740 INFO:teuthology.orchestra.run.smithi161.stdout:(6/8): openblas-0.3.15-6.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2024-04-27T04:25:01.982 INFO:teuthology.orchestra.run.smithi161.stdout:(7/8): python3-scipy-1.0.0-21.module+el8.9.0+20 39 MB/s | 14 MB 00:00 2024-04-27T04:25:02.324 INFO:teuthology.orchestra.run.smithi161.stdout:(8/8): ceph-mgr-diskprediction-local-18.2.2-125 7.0 MB/s | 7.4 MB 00:01 2024-04-27T04:25:02.325 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:25:02.325 INFO:teuthology.orchestra.run.smithi161.stdout:Total 34 MB/s | 36 MB 00:01 2024-04-27T04:25:02.570 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:25:02.589 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:25:02.589 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:25:02.901 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:25:02.909 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:25:03.455 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:25:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-27T04:25:03.767 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-27T04:25:03.926 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-27T04:25:04.584 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-27T04:25:04.772 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-27T04:25:05.381 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-27T04:25:05.606 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-27T04:25:06.299 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-27T04:25:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-27T04:25:06.855 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-27T04:25:08.614 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-27T04:25:09.903 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-27T04:25:10.113 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1258.g7d2137f 8/8 2024-04-27T04:25:10.415 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1258.g7d2137f 8/8 2024-04-27T04:25:10.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1258.g7d2137f 1/8 2024-04-27T04:25:10.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-27T04:25:10.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 3/8 2024-04-27T04:25:10.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-27T04:25:10.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-27T04:25:10.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 6/8 2024-04-27T04:25:10.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 7/8 2024-04-27T04:25:15.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 8/8 2024-04-27T04:25:15.235 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:25:15.792 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:15.792 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:25:15.792 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1258.g7d2137f0.el8.noarch 2024-04-27T04:25:15.792 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-27T04:25:15.792 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-27T04:25:15.792 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-27T04:25:15.793 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-27T04:25:15.793 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-27T04:25:15.793 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-27T04:25:15.793 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-27T04:25:15.793 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:15.793 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:25:15.958 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-rook 2024-04-27T04:25:16.312 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:25:17.083 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 115 kB/s | 4.1 kB 00:00 2024-04-27T04:25:17.331 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 134 kB/s | 4.5 kB 00:00 2024-04-27T04:25:19.920 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:25:19.921 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================== 2024-04-27T04:25:19.921 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-04-27T04:25:19.922 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================== 2024-04-27T04:25:19.922 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:25:19.922 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook noarch 2:18.2.2-1258.g7d2137f0.el8 ceph-noarch 49 k 2024-04-27T04:25:19.922 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-04-27T04:25:19.922 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-27T04:25:19.922 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-27T04:25:19.922 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-27T04:25:19.922 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-27T04:25:19.922 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-27T04:25:19.922 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-27T04:25:19.922 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-27T04:25:19.922 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-27T04:25:19.923 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-27T04:25:19.923 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-27T04:25:19.923 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-27T04:25:19.923 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-27T04:25:19.923 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-27T04:25:19.923 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:19.923 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:25:19.923 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================== 2024-04-27T04:25:19.923 INFO:teuthology.orchestra.run.smithi161.stdout:Install 14 Packages 2024-04-27T04:25:19.923 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:19.924 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 2.0 M 2024-04-27T04:25:19.924 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 25 M 2024-04-27T04:25:19.924 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:25:20.330 INFO:teuthology.orchestra.run.smithi161.stdout:(1/14): python3-cachetools-3.1.1-4.el8.noarch.r 143 kB/s | 33 kB 00:00 2024-04-27T04:25:20.356 INFO:teuthology.orchestra.run.smithi161.stdout:(2/14): python3-certifi-2018.10.15-7.el8.noarch 64 kB/s | 16 kB 00:00 2024-04-27T04:25:20.381 INFO:teuthology.orchestra.run.smithi161.stdout:(3/14): ceph-mgr-rook-18.2.2-1258.g7d2137f0.el8 175 kB/s | 49 kB 00:00 2024-04-27T04:25:20.431 INFO:teuthology.orchestra.run.smithi161.stdout:(4/14): python3-google-auth-1.1.1-10.el8.noarch 807 kB/s | 81 kB 00:00 2024-04-27T04:25:20.506 INFO:teuthology.orchestra.run.smithi161.stdout:(5/14): python3-websocket-client-0.56.0-5.el8.n 819 kB/s | 61 kB 00:00 2024-04-27T04:25:20.590 INFO:teuthology.orchestra.run.smithi161.stdout:(6/14): python3-rsa-4.9-2.el8.noarch.rpm 307 kB/s | 64 kB 00:00 2024-04-27T04:25:20.617 INFO:teuthology.orchestra.run.smithi161.stdout:(7/14): python3-jwt-1.6.1-2.el8.noarch.rpm 392 kB/s | 43 kB 00:00 2024-04-27T04:25:20.691 INFO:teuthology.orchestra.run.smithi161.stdout:(8/14): python3-kubernetes-11.0.0-6.el8.noarch. 3.5 MB/s | 1.2 MB 00:00 2024-04-27T04:25:20.716 INFO:teuthology.orchestra.run.smithi161.stdout:(9/14): python3-requests-oauthlib-1.0.0-1.el8.n 450 kB/s | 43 kB 00:00 2024-04-27T04:25:20.749 INFO:teuthology.orchestra.run.smithi161.stdout:(10/14): python3-oauthlib-2.1.0-1.el8.noarch.rp 973 kB/s | 155 kB 00:00 2024-04-27T04:25:20.775 INFO:teuthology.orchestra.run.smithi161.stdout:(11/14): python3-pyasn1-modules-0.3.7-6.el8.noa 2.0 MB/s | 110 kB 00:00 2024-04-27T04:25:20.800 INFO:teuthology.orchestra.run.smithi161.stdout:(12/14): python3-jsonpointer-1.10-11.el8.noarch 184 kB/s | 20 kB 00:00 2024-04-27T04:25:20.825 INFO:teuthology.orchestra.run.smithi161.stdout:(13/14): python3-jsonpatch-1.21-2.el8.noarch.rp 352 kB/s | 27 kB 00:00 2024-04-27T04:25:20.850 INFO:teuthology.orchestra.run.smithi161.stdout:(14/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.6 MB/s | 126 kB 00:00 2024-04-27T04:25:20.853 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:25:20.853 INFO:teuthology.orchestra.run.smithi161.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-04-27T04:25:20.920 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:25:20.935 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:25:20.935 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:25:21.018 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:25:21.026 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:25:21.572 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:25:21.829 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-27T04:25:22.055 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-27T04:25:22.259 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-27T04:25:22.443 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/14 2024-04-27T04:25:22.654 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/14 2024-04-27T04:25:22.906 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-04-27T04:25:23.207 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-04-27T04:25:23.464 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-04-27T04:25:23.709 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-27T04:25:23.909 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-27T04:25:24.167 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-27T04:25:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-27T04:25:26.247 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-27T04:25:26.448 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-rook-2:18.2.2-1258.g7d2137f0.el8.noarch 14/14 2024-04-27T04:25:26.761 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1258.g7d2137f0.el8.noarch 14/14 2024-04-27T04:25:26.761 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1258.g7d2137f0.el8.noarch 1/14 2024-04-27T04:25:26.761 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-27T04:25:26.761 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-27T04:25:26.761 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-27T04:25:26.761 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-27T04:25:26.761 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-27T04:25:26.762 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-27T04:25:26.762 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-04-27T04:25:26.762 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-04-27T04:25:26.762 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-04-27T04:25:26.762 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/14 2024-04-27T04:25:26.762 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/14 2024-04-27T04:25:26.762 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-04-27T04:25:31.702 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 14/14 2024-04-27T04:25:31.702 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:25:32.306 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:32.306 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:25:32.306 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook-2:18.2.2-1258.g7d2137f0.el8.noarch 2024-04-27T04:25:32.306 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-27T04:25:32.306 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-27T04:25:32.306 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-27T04:25:32.306 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-27T04:25:32.306 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-27T04:25:32.306 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-27T04:25:32.306 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-27T04:25:32.307 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-27T04:25:32.307 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-27T04:25:32.307 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-27T04:25:32.307 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-27T04:25:32.307 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-27T04:25:32.307 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-27T04:25:32.307 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:32.307 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:25:32.525 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-cephadm 2024-04-27T04:25:32.882 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:25:33.672 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 kB/s | 4.1 kB 00:00 2024-04-27T04:25:33.931 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 121 kB/s | 4.5 kB 00:00 2024-04-27T04:25:36.557 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:25:36.558 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================================== 2024-04-27T04:25:36.558 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-04-27T04:25:36.558 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================================== 2024-04-27T04:25:36.558 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:25:36.559 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1258.g7d2137f0.el8 ceph-noarch 138 k 2024-04-27T04:25:36.559 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-04-27T04:25:36.559 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-27T04:25:36.559 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-27T04:25:36.559 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-27T04:25:36.559 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-27T04:25:36.559 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:36.559 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:25:36.559 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================================== 2024-04-27T04:25:36.559 INFO:teuthology.orchestra.run.smithi161.stdout:Install 5 Packages 2024-04-27T04:25:36.559 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:36.560 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 6.0 M 2024-04-27T04:25:36.560 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 26 M 2024-04-27T04:25:36.560 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:25:36.886 INFO:teuthology.orchestra.run.smithi161.stdout:(1/5): ceph-mgr-cephadm-18.2.2-1258.g7d2137f0.e 733 kB/s | 138 kB 00:00 2024-04-27T04:25:37.045 INFO:teuthology.orchestra.run.smithi161.stdout:(2/5): python3-natsort-7.1.1-2.el8.noarch.rpm 173 kB/s | 60 kB 00:00 2024-04-27T04:25:37.287 INFO:teuthology.orchestra.run.smithi161.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 12 MB/s | 4.8 MB 00:00 2024-04-27T04:25:37.312 INFO:teuthology.orchestra.run.smithi161.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 775 kB/s | 476 kB 00:00 2024-04-27T04:25:37.345 INFO:teuthology.orchestra.run.smithi161.stdout:(5/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.7 MB/s | 538 kB 00:00 2024-04-27T04:25:37.346 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:25:37.347 INFO:teuthology.orchestra.run.smithi161.stdout:Total 7.6 MB/s | 6.0 MB 00:00 2024-04-27T04:25:37.402 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:25:37.413 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:25:37.413 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:25:37.504 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:25:37.513 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:25:38.467 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:25:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-27T04:25:39.085 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-27T04:25:39.398 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-27T04:25:39.667 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-27T04:25:39.892 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1258.g7d2137f0.el8.noarch 5/5 2024-04-27T04:25:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1258.g7d2137f0.el8.noarch 5/5 2024-04-27T04:25:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1258.g7d2137f0.el8.noarch 1/5 2024-04-27T04:25:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-27T04:25:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-27T04:25:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-27T04:25:45.835 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-27T04:25:45.835 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:25:46.411 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:46.411 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:25:46.411 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm-2:18.2.2-1258.g7d2137f0.el8.noarch 2024-04-27T04:25:46.411 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-27T04:25:46.411 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-27T04:25:46.411 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-27T04:25:46.411 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-27T04:25:46.411 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:46.411 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:25:46.568 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-fuse 2024-04-27T04:25:46.927 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:25:47.801 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 106 kB/s | 4.1 kB 00:00 2024-04-27T04:25:48.186 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-27T04:25:50.799 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:25:50.800 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:25:50.800 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-04-27T04:25:50.800 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:25:50.800 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:25:50.800 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse 2024-04-27T04:25:50.800 INFO:teuthology.orchestra.run.smithi161.stdout: x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 859 k 2024-04-27T04:25:50.800 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-04-27T04:25:50.800 INFO:teuthology.orchestra.run.smithi161.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-27T04:25:50.800 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:50.800 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:25:50.801 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:25:50.801 INFO:teuthology.orchestra.run.smithi161.stdout:Install 2 Packages 2024-04-27T04:25:50.801 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:50.801 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 942 k 2024-04-27T04:25:50.802 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 3.0 M 2024-04-27T04:25:50.802 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:25:51.055 INFO:teuthology.orchestra.run.smithi161.stdout:(1/2): ceph-fuse-18.2.2-1258.g7d2137f0.el8.x86_ 3.3 MB/s | 859 kB 00:00 2024-04-27T04:25:51.155 INFO:teuthology.orchestra.run.smithi161.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 235 kB/s | 83 kB 00:00 2024-04-27T04:25:51.155 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:25:51.155 INFO:teuthology.orchestra.run.smithi161.stdout:Total 2.6 MB/s | 942 kB 00:00 2024-04-27T04:25:51.167 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:25:51.189 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:25:51.189 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:25:51.274 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:25:51.283 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:25:51.466 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:25:51.723 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-27T04:25:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-2:18.2.2-1258.g7d2137f0.el8.x86_64 2/2 2024-04-27T04:25:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1258.g7d2137f0.el8.x86_64 2/2 2024-04-27T04:25:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/2 2024-04-27T04:25:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-27T04:25:57.435 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:25:57.993 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:57.993 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:25:57.993 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-2:18.2.2-1258.g7d2137f0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-27T04:25:57.994 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:25:57.994 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:25:58.564 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-volume 2024-04-27T04:25:58.927 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:25:59.682 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 109 kB/s | 4.1 kB 00:00 2024-04-27T04:25:59.940 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 122 kB/s | 4.5 kB 00:00 2024-04-27T04:26:02.586 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:26:02.586 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================== 2024-04-27T04:26:02.586 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-04-27T04:26:02.586 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================== 2024-04-27T04:26:02.587 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:26:02.587 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume noarch 2:18.2.2-1258.g7d2137f0.el8 ceph-noarch 263 k 2024-04-27T04:26:02.587 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-04-27T04:26:02.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-27T04:26:02.587 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:26:02.587 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:26:02.587 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================== 2024-04-27T04:26:02.587 INFO:teuthology.orchestra.run.smithi161.stdout:Install 2 Packages 2024-04-27T04:26:02.587 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:26:02.588 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 311 k 2024-04-27T04:26:02.588 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 1.4 M 2024-04-27T04:26:02.588 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:26:02.669 INFO:teuthology.orchestra.run.smithi161.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 591 kB/s | 48 kB 00:00 2024-04-27T04:26:02.974 INFO:teuthology.orchestra.run.smithi161.stdout:(2/2): ceph-volume-18.2.2-1258.g7d2137f0.el8.no 681 kB/s | 263 kB 00:00 2024-04-27T04:26:02.974 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:26:02.975 INFO:teuthology.orchestra.run.smithi161.stdout:Total 803 kB/s | 311 kB 00:00 2024-04-27T04:26:02.986 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:26:02.999 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:26:02.999 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:26:03.037 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:26:03.046 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:26:03.221 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:26:03.578 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-27T04:26:03.847 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-volume-2:18.2.2-1258.g7d2137f0.el8.noarch 2/2 2024-04-27T04:26:04.244 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-volume-2:18.2.2-1258.g7d2137f0.el8.noarch 2/2 2024-04-27T04:26:04.244 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-volume-2:18.2.2-1258.g7d2137f0.el8.noarch 1/2 2024-04-27T04:26:09.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-27T04:26:09.235 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:26:09.797 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:26:09.797 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:26:09.798 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume-2:18.2.2-1258.g7d2137f0.el8.noarch 2024-04-27T04:26:09.798 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-27T04:26:09.798 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:26:09.798 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:26:09.944 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librados-devel 2024-04-27T04:26:10.305 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:26:11.139 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 108 kB/s | 4.1 kB 00:00 2024-04-27T04:26:11.396 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 103 kB/s | 4.5 kB 00:00 2024-04-27T04:26:14.183 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:26:14.183 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:26:14.183 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-04-27T04:26:14.183 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:26:14.183 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:26:14.183 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 139 k 2024-04-27T04:26:14.183 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:26:14.183 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:26:14.184 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:26:14.184 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-04-27T04:26:14.184 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:26:14.184 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 139 k 2024-04-27T04:26:14.184 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 513 k 2024-04-27T04:26:14.184 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:26:14.368 INFO:teuthology.orchestra.run.smithi161.stdout:librados-devel-18.2.2-1258.g7d2137f0.el8.x86_64 756 kB/s | 139 kB 00:00 2024-04-27T04:26:14.369 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:26:14.369 INFO:teuthology.orchestra.run.smithi161.stdout:Total 751 kB/s | 139 kB 00:00 2024-04-27T04:26:14.369 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:26:14.375 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:26:14.375 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:26:14.433 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:26:14.442 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:26:14.581 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:26:14.785 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:26:15.059 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados-devel-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:26:26.536 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:26:26.536 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:26:27.101 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:26:27.101 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:26:27.101 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:26:27.101 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:26:27.102 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:26:27.255 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install libcephfs2 2024-04-27T04:26:27.614 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:26:29.538 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-27T04:26:29.874 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 kB/s | 4.5 kB 00:00 2024-04-27T04:26:32.287 INFO:teuthology.orchestra.run.smithi161.stdout:Package libcephfs2-2:18.2.2-1258.g7d2137f0.el8.x86_64 is already installed. 2024-04-27T04:26:32.475 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:26:32.477 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-04-27T04:26:32.477 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:26:32.618 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install libcephfs-devel 2024-04-27T04:26:32.979 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:26:35.104 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 kB/s | 4.1 kB 00:00 2024-04-27T04:26:35.404 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 56 kB/s | 4.5 kB 00:00 2024-04-27T04:26:38.006 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:26:38.006 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:26:38.006 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-04-27T04:26:38.007 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:26:38.007 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:26:38.007 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 31 k 2024-04-27T04:26:38.007 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:26:38.007 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:26:38.007 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:26:38.007 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-04-27T04:26:38.007 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:26:38.007 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 31 k 2024-04-27T04:26:38.007 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 139 k 2024-04-27T04:26:38.007 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:26:38.214 INFO:teuthology.orchestra.run.smithi161.stdout:libcephfs-devel-18.2.2-1258.g7d2137f0.el8.x86_6 149 kB/s | 31 kB 00:00 2024-04-27T04:26:38.215 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:26:38.215 INFO:teuthology.orchestra.run.smithi161.stdout:Total 148 kB/s | 31 kB 00:00 2024-04-27T04:26:38.215 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:26:38.219 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:26:38.219 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:26:38.231 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:26:38.240 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:26:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:26:38.506 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs-devel-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:26:38.750 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:26:46.750 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs-devel-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:26:46.750 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:26:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:26:47.303 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:26:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:26:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:26:47.303 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:26:47.465 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librados2 2024-04-27T04:26:47.826 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:26:48.836 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-27T04:26:49.175 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-04-27T04:26:51.706 INFO:teuthology.orchestra.run.smithi161.stdout:Package librados2-2:18.2.2-1258.g7d2137f0.el8.x86_64 is already installed. 2024-04-27T04:26:51.895 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:26:51.897 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-04-27T04:26:51.897 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:26:52.050 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librbd1 2024-04-27T04:26:52.411 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:26:53.374 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 kB/s | 4.1 kB 00:00 2024-04-27T04:26:53.779 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2024-04-27T04:26:56.303 INFO:teuthology.orchestra.run.smithi161.stdout:Package librbd1-2:18.2.2-1258.g7d2137f0.el8.x86_64 is already installed. 2024-04-27T04:26:56.490 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:26:56.492 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-04-27T04:26:56.492 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:26:56.641 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rados 2024-04-27T04:26:57.000 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:26:58.807 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-27T04:26:59.185 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-27T04:27:01.724 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-rados-2:18.2.2-1258.g7d2137f0.el8.x86_64 is already installed. 2024-04-27T04:27:01.912 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:27:01.913 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-04-27T04:27:01.913 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:27:02.073 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rgw 2024-04-27T04:27:02.432 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:27:04.332 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-27T04:27:04.679 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-27T04:27:07.231 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-rgw-2:18.2.2-1258.g7d2137f0.el8.x86_64 is already installed. 2024-04-27T04:27:07.418 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:27:07.419 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-04-27T04:27:07.419 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:27:07.556 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-cephfs 2024-04-27T04:27:07.917 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:27:09.624 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-27T04:27:09.999 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-27T04:27:12.524 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-cephfs-2:18.2.2-1258.g7d2137f0.el8.x86_64 is already installed. 2024-04-27T04:27:12.715 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:27:12.716 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-04-27T04:27:12.716 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:27:12.851 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rbd 2024-04-27T04:27:13.212 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:27:14.955 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-27T04:27:15.348 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2024-04-27T04:27:17.744 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-rbd-2:18.2.2-1258.g7d2137f0.el8.x86_64 is already installed. 2024-04-27T04:27:17.934 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:27:17.935 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-04-27T04:27:17.935 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:27:18.070 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-fuse 2024-04-27T04:27:18.428 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:27:20.222 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 kB/s | 4.1 kB 00:00 2024-04-27T04:27:20.592 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-27T04:27:23.199 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:27:23.200 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:27:23.200 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2024-04-27T04:27:23.200 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:27:23.200 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:27:23.200 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 95 k 2024-04-27T04:27:23.200 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:27:23.200 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:27:23.200 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:27:23.200 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-04-27T04:27:23.201 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:27:23.201 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 95 k 2024-04-27T04:27:23.201 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 280 k 2024-04-27T04:27:23.201 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:27:23.386 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-fuse-18.2.2-1258.g7d2137f0.el8.x86_64.rpm 515 kB/s | 95 kB 00:00 2024-04-27T04:27:23.386 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:27:23.386 INFO:teuthology.orchestra.run.smithi161.stdout:Total 511 kB/s | 95 kB 00:00 2024-04-27T04:27:23.387 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:27:23.397 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:27:23.397 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:27:23.453 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:27:23.462 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:27:23.575 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:27:23.750 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:27:24.051 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:27:31.447 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:27:31.447 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:27:32.004 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:27:32.005 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:27:32.005 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:27:32.005 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:27:32.005 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:27:32.165 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-mirror 2024-04-27T04:27:32.528 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:27:34.255 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 kB/s | 4.1 kB 00:00 2024-04-27T04:27:34.659 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-27T04:27:37.245 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:27:37.245 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:27:37.246 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-04-27T04:27:37.246 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:27:37.246 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:27:37.246 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 3.5 M 2024-04-27T04:27:37.246 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:27:37.246 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:27:37.246 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:27:37.246 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-04-27T04:27:37.246 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:27:37.247 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 3.5 M 2024-04-27T04:27:37.247 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 16 M 2024-04-27T04:27:37.247 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:27:37.764 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-mirror-18.2.2-1258.g7d2137f0.el8.x86_64.rpm 6.7 MB/s | 3.5 MB 00:00 2024-04-27T04:27:37.765 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:27:37.765 INFO:teuthology.orchestra.run.smithi161.stdout:Total 6.7 MB/s | 3.5 MB 00:00 2024-04-27T04:27:37.765 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:27:37.778 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:27:37.778 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:27:37.847 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:27:37.855 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:27:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:27:38.491 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:27:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:27:49.447 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:27:49.447 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:27:50.025 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:27:50.025 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:27:50.026 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:27:50.026 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:27:50.026 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:27:50.188 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-nbd 2024-04-27T04:27:50.558 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2024-04-27T04:27:51.480 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 108 kB/s | 4.1 kB 00:00 2024-04-27T04:27:51.822 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 105 kB/s | 4.5 kB 00:00 2024-04-27T04:27:54.438 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-27T04:27:54.439 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:27:54.439 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2024-04-27T04:27:54.439 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:27:54.439 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-27T04:27:54.439 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd x86_64 2:18.2.2-1258.g7d2137f0.el8 ceph 182 k 2024-04-27T04:27:54.439 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:27:54.439 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-27T04:27:54.439 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-04-27T04:27:54.439 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-04-27T04:27:54.440 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:27:54.440 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 182 k 2024-04-27T04:27:54.440 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 572 k 2024-04-27T04:27:54.440 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-27T04:27:54.667 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-nbd-18.2.2-1258.g7d2137f0.el8.x86_64.rpm 803 kB/s | 182 kB 00:00 2024-04-27T04:27:54.667 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-27T04:27:54.667 INFO:teuthology.orchestra.run.smithi161.stdout:Total 798 kB/s | 182 kB 00:00 2024-04-27T04:27:54.668 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-27T04:27:54.679 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-27T04:27:54.679 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-27T04:27:54.735 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-27T04:27:54.743 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-27T04:27:54.863 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-27T04:27:55.062 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:27:55.337 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:28:03.891 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-2:18.2.2-1258.g7d2137f0.el8.x86_64 1/1 2024-04-27T04:28:03.892 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2024-04-27T04:28:04.460 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:04.460 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-27T04:28:04.461 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-2:18.2.2-1258.g7d2137f0.el8.x86_64 2024-04-27T04:28:04.461 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:04.461 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-27T04:28:04.629 DEBUG:teuthology.parallel:result is None 2024-04-27T04:28:04.630 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7d2137f0a666c92a9411c1af5c33e4968be54da6 2024-04-27T04:28:04.769 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-27T04:28:04.792 INFO:teuthology.orchestra.run.smithi161.stdout:18.2.2-1258.g7d2137f0.el8 2024-04-27T04:28:04.792 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1258.g7d2137f0.el8 2024-04-27T04:28:04.793 INFO:teuthology.task.install:The correct ceph version 18.2.2-1258.g7d2137f0 is installed. 2024-04-27T04:28:04.795 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-27T04:28:04.795 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:04.796 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-27T04:28:04.860 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-27T04:28:04.860 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:04.860 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2024-04-27T04:28:04.926 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-27T04:28:04.993 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-27T04:28:04.993 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:04.994 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-27T04:28:05.060 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-27T04:28:05.126 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-27T04:28:05.126 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:05.126 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/stdin-killer 2024-04-27T04:28:05.193 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-27T04:28:05.259 INFO:teuthology.run_tasks:Running task ceph... 2024-04-27T04:28:05.334 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-27T04:28:05.334 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /var/log/ceph 2024-04-27T04:28:05.360 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-27T04:28:05.360 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-27T04:28:05.427 INFO:tasks.ceph:Creating extra log directories... 2024-04-27T04:28:05.427 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-27T04:28:05.500 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-27T04:28:05.500 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'setgroup': 'ceph', 'setuser': 'ceph', 'storage classes': 'LUKEWARM, FROZEN'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-27T04:28:05.501 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-26_22:40:03-rgw-reef-distro-default-smithi/7675694', 'branch': 'reef', 'description': 'rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7675694', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-26_22:40:03-rgw-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 1, 'size': 10}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'setgroup': 'ceph', 'setuser': 'ceph', 'storage classes': 'LUKEWARM, FROZEN'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '7d2137f0a666c92a9411c1af5c33e4968be54da6', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7d2137f0a666c92a9411c1af5c33e4968be54da6'}}, 'rgw': {'frontend': 'beast', 'storage classes': 'LUKEWARM, FROZEN'}, 's3tests': {'force-branch': 'ceph-reef'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef', 'sha1': '5403a02127f27ec6c3309ef057522ab70cdeb9d2'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'osd.2', 'mgr.0', 'client.0']], 'seed': 4225, 'sha1': '7d2137f0a666c92a9411c1af5c33e4968be54da6', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'rgw', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_5403a02127f27ec6c3309ef057522ab70cdeb9d2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '5403a02127f27ec6c3309ef057522ab70cdeb9d2', 'targets': {'smithi161.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAPGQNyqJD15fzyQVwadUa2FwDkQemrVVQxFyYqU2MnI4Jl0omGnJaswkXZsnzsgOs5w6pf+Ebgd2mmMp+O+Xec='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'rgw': ['client.0']}, {'tox': ['client.0']}, {'s3tests': {'client.0': {'extra_attrs': ['lifecycle'], 'lc_debug_interval': 10, 'rgw_server': 'client.0', 'storage classes': 'LUKEWARM, FROZEN'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-26_22:40:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-27T04:28:05.501 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-27T04:28:05.551 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/run/ceph 2024-04-27T04:28:05.618 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:05.618 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T04:28:05.675 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T04:28:05.676 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 2024-04-27T04:28:05.733 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T04:28:05.733 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T04:28:05.733 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 6h/6d Inode: 177490 Links: 1 2024-04-27T04:28:05.733 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T04:28:05.733 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T04:28:05.733 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-04-27 04:27:38.662578415 +0000 2024-04-27T04:28:05.733 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-04-27 04:21:13.854920712 +0000 2024-04-27T04:28:05.733 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-04-27 04:21:13.854920712 +0000 2024-04-27T04:28:05.733 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2024-04-27T04:28:05.733 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T04:28:05.798 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-04-27T04:28:05.798 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-04-27T04:28:05.799 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.00018038 s, 2.8 MB/s 2024-04-27T04:28:05.799 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T04:28:05.857 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 2024-04-27T04:28:05.915 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T04:28:05.915 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T04:28:05.915 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 6h/6d Inode: 181568 Links: 1 2024-04-27T04:28:05.915 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T04:28:05.915 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T04:28:05.915 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-04-27 04:27:38.662578415 +0000 2024-04-27T04:28:05.915 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-04-27 04:21:14.252919629 +0000 2024-04-27T04:28:05.915 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-04-27 04:21:14.252919629 +0000 2024-04-27T04:28:05.915 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2024-04-27T04:28:05.915 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T04:28:05.979 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-04-27T04:28:05.980 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-04-27T04:28:05.980 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000244135 s, 2.1 MB/s 2024-04-27T04:28:05.981 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T04:28:06.038 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 2024-04-27T04:28:06.095 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T04:28:06.095 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T04:28:06.095 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 6h/6d Inode: 184387 Links: 1 2024-04-27T04:28:06.095 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T04:28:06.095 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T04:28:06.096 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-04-27 04:27:38.662578415 +0000 2024-04-27T04:28:06.096 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-04-27 04:21:14.636918585 +0000 2024-04-27T04:28:06.096 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-04-27 04:21:14.636918585 +0000 2024-04-27T04:28:06.096 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2024-04-27T04:28:06.096 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T04:28:06.161 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-04-27T04:28:06.161 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-04-27T04:28:06.161 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000190851 s, 2.7 MB/s 2024-04-27T04:28:06.162 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T04:28:06.220 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 2024-04-27T04:28:06.278 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T04:28:06.278 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T04:28:06.278 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 6h/6d Inode: 182467 Links: 1 2024-04-27T04:28:06.278 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T04:28:06.278 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T04:28:06.278 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-04-27 04:27:38.662578415 +0000 2024-04-27T04:28:06.278 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-04-27 04:21:15.014917556 +0000 2024-04-27T04:28:06.278 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-04-27 04:21:15.014917556 +0000 2024-04-27T04:28:06.278 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2024-04-27T04:28:06.279 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T04:28:06.342 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-04-27T04:28:06.343 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-04-27T04:28:06.343 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000230663 s, 2.2 MB/s 2024-04-27T04:28:06.344 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T04:28:06.401 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-27T04:28:06.401 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-27T04:28:06.401 INFO:tasks.ceph:Generating config... 2024-04-27T04:28:06.403 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-27T04:28:06.403 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-27T04:28:06.403 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-27T04:28:06.403 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-04-27T04:28:06.403 INFO:tasks.ceph:[client] rgw lc debug interval = 10 2024-04-27T04:28:06.403 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-27T04:28:06.403 INFO:tasks.ceph:[client] setuser = ceph 2024-04-27T04:28:06.403 INFO:tasks.ceph:[client] storage classes = LUKEWARM, FROZEN 2024-04-27T04:28:06.403 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-27T04:28:06.403 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-27T04:28:06.403 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-27T04:28:06.403 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-27T04:28:06.403 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-27T04:28:06.404 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-27T04:28:06.404 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-27T04:28:06.404 INFO:tasks.ceph:Setting up mon.a... 2024-04-27T04:28:06.404 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-27T04:28:06.502 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/ceph.keyring 2024-04-27T04:28:06.505 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-27T04:28:06.550 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-27T04:28:06.577 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.161')] 2024-04-27T04:28:06.578 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.161'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'setgroup': 'ceph', 'setuser': 'ceph', 'storage classes': 'LUKEWARM, FROZEN'}, 'mon.a': {}} 2024-04-27T04:28:06.578 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:06.578 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-27T04:28:06.635 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.161 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-27T04:28:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:06.716 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-27T04:28:06.716 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: generated fsid acaf0272-2637-4013-a083-8ec5702e3ecd 2024-04-27T04:28:06.716 INFO:teuthology.orchestra.run.smithi161.stdout:setting min_mon_release = pacific 2024-04-27T04:28:06.716 INFO:teuthology.orchestra.run.smithi161.stdout:epoch 0 2024-04-27T04:28:06.716 INFO:teuthology.orchestra.run.smithi161.stdout:fsid acaf0272-2637-4013-a083-8ec5702e3ecd 2024-04-27T04:28:06.716 INFO:teuthology.orchestra.run.smithi161.stdout:last_changed 2024-04-27T04:28:06.717577+0000 2024-04-27T04:28:06.716 INFO:teuthology.orchestra.run.smithi161.stdout:created 2024-04-27T04:28:06.717577+0000 2024-04-27T04:28:06.716 INFO:teuthology.orchestra.run.smithi161.stdout:min_mon_release 16 (pacific) 2024-04-27T04:28:06.716 INFO:teuthology.orchestra.run.smithi161.stdout:election_strategy: 1 2024-04-27T04:28:06.717 INFO:teuthology.orchestra.run.smithi161.stdout:0: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.a 2024-04-27T04:28:06.717 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-27T04:28:06.718 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-27T04:28:06.774 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID acaf0272-2637-4013-a083-8ec5702e3ecd... 2024-04-27T04:28:06.775 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-27T04:28:06.860 INFO:teuthology.orchestra.run.smithi161.stdout:[global] 2024-04-27T04:28:06.860 INFO:teuthology.orchestra.run.smithi161.stdout: chdir = "" 2024-04-27T04:28:06.860 INFO:teuthology.orchestra.run.smithi161.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-27T04:28:06.860 INFO:teuthology.orchestra.run.smithi161.stdout: auth supported = cephx 2024-04-27T04:28:06.860 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: filestore xattr use omap = true 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: mon clock drift allowed = 1.000 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: osd crush chooseleaf type = 0 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: auth debug = true 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on old message = true 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on bug = true 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: mon pg warn max object skew = 0 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: # disable pg_autoscaler by default for new pools 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-27T04:28:06.861 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default size = 2 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow primary affinity = true 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow pg remap = true 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on legacy crush tunables = false 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on crush straw calc version zero = false 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on no sortbitwise = false 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on osd down out interval zero = false 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on too few osds = false 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: mon_allow_pool_size_one = true 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.862 INFO:teuthology.orchestra.run.smithi161.stdout: osd default data pool replay window = 5 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: mon allow pool delete = true 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: mon cluster log file level = debug 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: debug asserts on shutdown = true 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: mon health detail to clog = false 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: mon host = 172.21.15.161 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: fsid = acaf0272-2637-4013-a083-8ec5702e3ecd 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout:[osd] 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: osd journal size = 100 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub load threshold = 5.0 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub max interval = 600 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: osd mclock profile = high_recovery_ops 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.863 INFO:teuthology.orchestra.run.smithi161.stdout: osd recover clone overlap = true 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: osd recovery max chunk = 1048576 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug shutdown = true 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug op order = true 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify stray on activate = true 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: osd open classes on start = true 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug pg log writeout = true 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: osd deep scrub update digest min age = 30 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: osd map max advance = 10 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: journal zero on create = true 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: filestore ondisk finisher threads = 3 2024-04-27T04:28:06.864 INFO:teuthology.orchestra.run.smithi161.stdout: filestore apply finisher threads = 3 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: bdev debug aio = true 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug misdirected ops = true 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: debug osd = 20 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr] 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: debug mgr = 20 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: mgr/telemetry/nag = false 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout:[mon] 2024-04-27T04:28:06.865 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: debug paxos = 20 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: mon data avail warn = 5 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: mon mgr mkfs grace = 240 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd reporter subtree level = osd 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd prime pg temp = true 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: auth mon ticket ttl = 660 # 11m 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: auth service ticket ttl = 240 # 4m 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.866 INFO:teuthology.orchestra.run.smithi161.stdout: # don't complain about insecure global_id in the test suite 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: # 1m isn't quite enough 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: mon_down_mkfs_grace = 2m 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_filestore_osds = false 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout:[client] 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: rgw cache enabled = true 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable ops log = true 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable usage log = true 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-27T04:28:06.867 INFO:teuthology.orchestra.run.smithi161.stdout: debug rgw = 20 2024-04-27T04:28:06.868 INFO:teuthology.orchestra.run.smithi161.stdout: rgw crypt require ssl = False 2024-04-27T04:28:06.868 INFO:teuthology.orchestra.run.smithi161.stdout: rgw crypt s3 kms backend = testing 2024-04-27T04:28:06.868 INFO:teuthology.orchestra.run.smithi161.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-04-27T04:28:06.868 INFO:teuthology.orchestra.run.smithi161.stdout: rgw lc debug interval = 10 2024-04-27T04:28:06.868 INFO:teuthology.orchestra.run.smithi161.stdout: setgroup = ceph 2024-04-27T04:28:06.868 INFO:teuthology.orchestra.run.smithi161.stdout: setuser = ceph 2024-04-27T04:28:06.868 INFO:teuthology.orchestra.run.smithi161.stdout: storage classes = "LUKEWARM, FROZEN" 2024-04-27T04:28:06.868 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.a] 2024-04-27T04:28:06.871 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-27T04:28:06.872 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-27T04:28:06.969 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-27T04:28:06.969 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:06.969 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-27T04:28:06.986 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:06.986 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-27T04:28:07.043 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi161.front.sepia.ceph.com 2024-04-27T04:28:07.043 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:07.043 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-27T04:28:07.044 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-27T04:28:07.120 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:07.120 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-27T04:28:07.176 INFO:tasks.ceph:Setting up mon nodes... 2024-04-27T04:28:07.176 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-27T04:28:07.176 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mgr/ceph-0 && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.0 /var/lib/ceph/mgr/ceph-0/keyring 2024-04-27T04:28:07.270 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mgr/ceph-0/keyring 2024-04-27T04:28:07.273 INFO:tasks.ceph:Setting up mds nodes... 2024-04-27T04:28:07.273 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-27T04:28:07.273 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-27T04:28:07.316 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-27T04:28:07.329 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-27T04:28:07.329 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-27T04:28:07.329 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-27T04:28:07.395 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-27T04:28:07.395 INFO:tasks.ceph:role: osd.0 2024-04-27T04:28:07.395 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com 2024-04-27T04:28:07.395 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-27T04:28:07.469 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-27T04:28:07.470 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-27T04:28:07.470 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-27T04:28:07.470 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-27T04:28:07.470 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-27T04:28:07.470 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-04-27T04:28:07.470 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-27T04:28:07.470 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-27T04:28:07.470 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-27T04:28:07.470 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-27T04:28:07.893 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-04-27T04:28:07.895 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-04-27T04:28:07.895 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-27T04:28:08.078 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-27T04:28:08.112 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-27T04:28:08.181 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-27T04:28:08.181 INFO:tasks.ceph:role: osd.1 2024-04-27T04:28:08.181 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com 2024-04-27T04:28:08.181 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-27T04:28:08.254 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-27T04:28:08.254 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-27T04:28:08.255 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-27T04:28:08.255 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-27T04:28:08.255 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-27T04:28:08.255 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-04-27T04:28:08.255 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-27T04:28:08.255 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-27T04:28:08.255 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-27T04:28:08.255 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-27T04:28:08.677 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-04-27T04:28:08.679 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-04-27T04:28:08.679 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-27T04:28:08.795 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-27T04:28:08.826 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-27T04:28:08.893 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-27T04:28:08.893 INFO:tasks.ceph:role: osd.2 2024-04-27T04:28:08.893 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com 2024-04-27T04:28:08.893 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-27T04:28:08.967 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-27T04:28:08.967 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-27T04:28:08.967 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-27T04:28:08.967 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-27T04:28:08.967 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-27T04:28:08.967 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-04-27T04:28:08.967 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-27T04:28:08.967 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-27T04:28:08.967 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-27T04:28:08.967 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-27T04:28:09.406 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-04-27T04:28:09.408 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-04-27T04:28:09.408 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-27T04:28:09.555 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-27T04:28:09.589 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-27T04:28:09.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:09.678+0000 7f18e7b277c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-27T04:28:09.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:09.679+0000 7f18e7b277c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-27T04:28:09.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:09.679+0000 7f18e7b277c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-27T04:28:09.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:09.679+0000 7f18e7b277c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-27T04:28:09.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:09.679+0000 7f18e7b277c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-27T04:28:09.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:09.685+0000 7f18e7b277c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-27T04:28:11.773 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-27T04:28:11.805 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-27T04:28:11.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:11.898+0000 7f4445b6f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-27T04:28:11.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:11.898+0000 7f4445b6f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-27T04:28:11.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:11.898+0000 7f4445b6f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-27T04:28:11.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:11.898+0000 7f4445b6f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-27T04:28:11.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:11.898+0000 7f4445b6f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-27T04:28:11.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:11.899+0000 7f4445b6f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-27T04:28:13.988 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-27T04:28:14.020 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-27T04:28:14.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:14.114+0000 7fc13c0d77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-27T04:28:14.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:14.114+0000 7fc13c0d77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-27T04:28:14.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:14.114+0000 7fc13c0d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-27T04:28:14.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:14.114+0000 7fc13c0d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-27T04:28:14.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:14.114+0000 7fc13c0d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-27T04:28:14.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:14.115+0000 7fc13c0d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-27T04:28:16.191 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-27T04:28:16.223 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-27T04:28:16.223 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:16.223 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mgr/ceph-0/keyring of=/dev/stdout 2024-04-27T04:28:16.293 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:16.293 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-27T04:28:16.363 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:16.363 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-27T04:28:16.432 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:16.432 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-27T04:28:16.501 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:16.501 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-27T04:28:16.559 INFO:tasks.ceph:Adding keys to all mons... 2024-04-27T04:28:16.560 DEBUG:teuthology.orchestra.run.smithi161:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-27T04:28:16.626 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr.0] 2024-04-27T04:28:16.626 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQDXfixmrMA0EBAAk3BzP/VorVHVrNH68nwUrw== 2024-04-27T04:28:16.626 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.0] 2024-04-27T04:28:16.626 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQDZfixm7t53KBAAPgs85/izbtNkgmKXszlPgg== 2024-04-27T04:28:16.626 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.1] 2024-04-27T04:28:16.626 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQDbfixmKDeYNRAAJyNhSq3ESRC2dTCFOO0xag== 2024-04-27T04:28:16.626 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.2] 2024-04-27T04:28:16.626 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQDefixmjt3bBhAAno8Y2KuOXD306BXwgqeYNg== 2024-04-27T04:28:16.626 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-04-27T04:28:16.626 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQDXfixmsn/sEhAAygts8qB9Kqw2EJoQqUKj3A== 2024-04-27T04:28:16.627 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.0 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-27T04:28:16.723 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-27T04:28:16.777 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-27T04:28:16.830 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-27T04:28:16.884 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-27T04:28:16.940 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-27T04:28:16.940 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-27T04:28:16.966 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-27T04:28:17.065 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-27T04:28:17.092 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-27T04:28:17.149 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-27T04:28:17.149 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-27T04:28:17.150 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-27T04:28:17.192 INFO:tasks.ceph.mon.a:Started 2024-04-27T04:28:17.192 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-27T04:28:17.192 INFO:tasks.ceph.mgr.0:Restarting daemon 2024-04-27T04:28:17.192 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i 0 2024-04-27T04:28:17.194 INFO:tasks.ceph.mgr.0:Started 2024-04-27T04:28:17.194 INFO:tasks.ceph:Setting crush tunables to default 2024-04-27T04:28:17.194 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd crush tunables default 2024-04-27T04:28:17.419 INFO:teuthology.orchestra.run.smithi161.stderr:adjusted tunables profile to default 2024-04-27T04:28:17.429 INFO:tasks.ceph:check_enable_crimson: False 2024-04-27T04:28:17.430 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-27T04:28:17.430 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:17.430 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-27T04:28:17.456 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:17.456 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-27T04:28:17.523 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:17.524 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-27T04:28:17.591 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 2c03baab-c9f4-452c-9c03-407aefaa7556 0 2024-04-27T04:28:17.844 INFO:teuthology.orchestra.run.smithi161.stdout:0 2024-04-27T04:28:17.857 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 1d2b9892-a981-4801-bb85-7f2006509d26 1 2024-04-27T04:28:17.927 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:17.927+0000 7fd54b999200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-27T04:28:18.012 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:18.012+0000 7fd54b999200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-27T04:28:18.073 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-04-27T04:28:18.090 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 9d0ae2f2-18b8-4349-abba-bfa054f968ef 2 2024-04-27T04:28:18.094 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:18.094+0000 7fd54b999200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-27T04:28:18.169 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:18.169+0000 7fd54b999200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-27T04:28:18.308 INFO:teuthology.orchestra.run.smithi161.stdout:2 2024-04-27T04:28:18.321 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-27T04:28:18.321 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-27T04:28:18.352 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:18.352+0000 7fd54b999200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-27T04:28:18.354 INFO:tasks.ceph.osd.0:Started 2024-04-27T04:28:18.354 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-27T04:28:18.354 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-27T04:28:18.356 INFO:tasks.ceph.osd.1:Started 2024-04-27T04:28:18.356 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-27T04:28:18.356 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-27T04:28:18.358 INFO:tasks.ceph.osd.2:Started 2024-04-27T04:28:18.358 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T04:28:18.546 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:18.546+0000 7fd54b999200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-27T04:28:18.586 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:18.586 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":5,"fsid":"acaf0272-2637-4013-a083-8ec5702e3ecd","created":"2024-04-27T04:28:17.321378+0000","modified":"2024-04-27T04:28:18.306825+0000","last_up_change":"0.000000","last_in_change":"2024-04-27T04:28:18.306825+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c03baab-c9f4-452c-9c03-407aefaa7556","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d2b9892-a981-4801-bb85-7f2006509d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d0ae2f2-18b8-4349-abba-bfa054f968ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T04:28:18.600 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-27T04:28:18.601 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-27T04:28:18.662 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:18.662+0000 7fd54b999200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-27T04:28:18.757 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:18.757+0000 7fd54b999200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-27T04:28:18.832 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:18.832+0000 7fd54b999200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-27T04:28:18.902 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-27T04:28:18.964 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-04-27T04:28:18.964+0000 7f49388aa7c0 -1 Falling back to public interface 2024-04-27T04:28:18.965 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-04-27T04:28:18.965+0000 7f9acdb3a7c0 -1 Falling back to public interface 2024-04-27T04:28:18.965 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-04-27T04:28:18.965+0000 7f492fc68700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6805/0: (98) Address already in use 2024-04-27T04:28:18.965 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-04-27T04:28:18.966+0000 7f492fc68700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6812/0: (98) Address already in use 2024-04-27T04:28:18.977 INFO:tasks.ceph.osd.1.smithi161.stderr:2024-04-27T04:28:18.977+0000 7f439c5927c0 -1 Falling back to public interface 2024-04-27T04:28:19.073 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:19.073+0000 7fd54b999200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-27T04:28:19.123 INFO:teuthology.misc.health.smithi161.stdout: 2024-04-27T04:28:19.123 INFO:teuthology.misc.health.smithi161.stdout:{"epoch":5,"fsid":"acaf0272-2637-4013-a083-8ec5702e3ecd","created":"2024-04-27T04:28:17.321378+0000","modified":"2024-04-27T04:28:18.306825+0000","last_up_change":"0.000000","last_in_change":"2024-04-27T04:28:18.306825+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c03baab-c9f4-452c-9c03-407aefaa7556","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d2b9892-a981-4801-bb85-7f2006509d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d0ae2f2-18b8-4349-abba-bfa054f968ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T04:28:19.134 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-27T04:28:19.147 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:19.148+0000 7fd54b999200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-27T04:28:19.523 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:19.523+0000 7fd54b999200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-27T04:28:19.847 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:19.847+0000 7fd54b999200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-27T04:28:20.006 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:20.006+0000 7fd54b999200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-27T04:28:20.086 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:20.086+0000 7fd54b999200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-27T04:28:20.787 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:20.787+0000 7fd54b999200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-27T04:28:20.869 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:20.869+0000 7fd54b999200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-27T04:28:21.003 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:21.004+0000 7fd54b999200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-27T04:28:21.204 INFO:tasks.ceph.osd.1.smithi161.stderr:2024-04-27T04:28:21.204+0000 7f439c5927c0 -1 osd.1 0 log_to_monitors true 2024-04-27T04:28:21.561 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:21.561+0000 7fd54b999200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-27T04:28:21.721 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-04-27T04:28:21.721+0000 7f49388aa7c0 -1 osd.0 0 log_to_monitors true 2024-04-27T04:28:21.722 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-04-27T04:28:21.722+0000 7f9acdb3a7c0 -1 osd.2 0 log_to_monitors true 2024-04-27T04:28:21.744 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:21.744+0000 7fd54b999200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-27T04:28:21.908 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:21.908+0000 7fd54b999200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-27T04:28:22.767 INFO:tasks.ceph.osd.1.smithi161.stderr:2024-04-27T04:28:22.767+0000 7f439094a700 -1 osd.1 0 waiting for initial osdmap 2024-04-27T04:28:22.770 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:22.771+0000 7fd54b999200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-27T04:28:22.791 INFO:tasks.ceph.osd.1.smithi161.stderr:2024-04-27T04:28:22.792+0000 7f4388524700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T04:28:23.010 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:23.010+0000 7fd54b999200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-27T04:28:23.084 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:23.084+0000 7fd54b999200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-27T04:28:23.490 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:23.490+0000 7fd54b999200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-27T04:28:24.092 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-04-27T04:28:24.092+0000 7f9ac1ef2700 -1 osd.2 0 waiting for initial osdmap 2024-04-27T04:28:24.196 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-04-27T04:28:24.196+0000 7f9ab9acc700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T04:28:24.553 INFO:tasks.ceph.mgr.0.smithi161.stderr:2024-04-27T04:28:24.553+0000 7fd506c13700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-27T04:28:24.874 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-04-27T04:28:24.874+0000 7f492d463700 -1 osd.0 0 waiting for initial osdmap 2024-04-27T04:28:24.885 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-04-27T04:28:24.885+0000 7f492483c700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T04:28:25.437 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-27T04:28:25.767 INFO:teuthology.misc.health.smithi161.stdout: 2024-04-27T04:28:25.767 INFO:teuthology.misc.health.smithi161.stdout:{"epoch":9,"fsid":"acaf0272-2637-4013-a083-8ec5702e3ecd","created":"2024-04-27T04:28:17.321378+0000","modified":"2024-04-27T04:28:25.094539+0000","last_up_change":"2024-04-27T04:28:25.094539+0000","last_in_change":"2024-04-27T04:28:18.306825+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c03baab-c9f4-452c-9c03-407aefaa7556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":40817},{"type":"v1","addr":"172.21.15.161:6801","nonce":40817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":40817},{"type":"v1","addr":"172.21.15.161:6803","nonce":40817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":40817},{"type":"v1","addr":"172.21.15.161:6813","nonce":40817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":40817},{"type":"v1","addr":"172.21.15.161:6808","nonce":40817}]},"public_addr":"172.21.15.161:6801/40817","cluster_addr":"172.21.15.161:6803/40817","heartbeat_back_addr":"172.21.15.161:6813/40817","heartbeat_front_addr":"172.21.15.161:6808/40817","state":["exists","up"]},{"osd":1,"uuid":"1d2b9892-a981-4801-bb85-7f2006509d26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":40831},{"type":"v1","addr":"172.21.15.161:6817","nonce":40831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":40831},{"type":"v1","addr":"172.21.15.161:6819","nonce":40831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":40831},{"type":"v1","addr":"172.21.15.161:6823","nonce":40831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":40831},{"type":"v1","addr":"172.21.15.161:6821","nonce":40831}]},"public_addr":"172.21.15.161:6817/40831","cluster_addr":"172.21.15.161:6819/40831","heartbeat_back_addr":"172.21.15.161:6823/40831","heartbeat_front_addr":"172.21.15.161:6821/40831","state":["exists","up"]},{"osd":2,"uuid":"9d0ae2f2-18b8-4349-abba-bfa054f968ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":40818},{"type":"v1","addr":"172.21.15.161:6805","nonce":40818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6807","nonce":40818},{"type":"v1","addr":"172.21.15.161:6809","nonce":40818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":40818},{"type":"v1","addr":"172.21.15.161:6815","nonce":40818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6811","nonce":40818},{"type":"v1","addr":"172.21.15.161:6812","nonce":40818}]},"public_addr":"172.21.15.161:6805/40818","cluster_addr":"172.21.15.161:6809/40818","heartbeat_back_addr":"172.21.15.161:6815/40818","heartbeat_front_addr":"172.21.15.161:6812/40818","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T04:28:22.207840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T04:28:25.780 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-27T04:28:25.780 INFO:tasks.ceph:Creating RBD pool 2024-04-27T04:28:25.780 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-27T04:28:26.559 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'rbd' created 2024-04-27T04:28:26.571 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-27T04:28:27.566 INFO:teuthology.orchestra.run.smithi161.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-27T04:28:27.580 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-27T04:28:27.580 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-27T04:28:27.580 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-27T04:28:27.580 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-27T04:28:27.580 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-27T04:28:28.021 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:28.035 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"0","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":40437},{"type":"v1","addr":"172.21.15.161:6825","nonce":40437}]},"active_addr":"172.21.15.161:6825/40437","active_change":"2024-04-27T04:28:23.503786+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":3778254258}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":892161744}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":3660840090}]}]} 2024-04-27T04:28:28.037 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-27T04:28:28.037 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-27T04:28:28.037 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T04:28:28.366 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:28.367 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":11,"fsid":"acaf0272-2637-4013-a083-8ec5702e3ecd","created":"2024-04-27T04:28:17.321378+0000","modified":"2024-04-27T04:28:27.553413+0000","last_up_change":"2024-04-27T04:28:25.094539+0000","last_in_change":"2024-04-27T04:28:18.306825+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T04:28:25.592985+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-27T04:28:26.125142+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c03baab-c9f4-452c-9c03-407aefaa7556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":40817},{"type":"v1","addr":"172.21.15.161:6801","nonce":40817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":40817},{"type":"v1","addr":"172.21.15.161:6803","nonce":40817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":40817},{"type":"v1","addr":"172.21.15.161:6813","nonce":40817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":40817},{"type":"v1","addr":"172.21.15.161:6808","nonce":40817}]},"public_addr":"172.21.15.161:6801/40817","cluster_addr":"172.21.15.161:6803/40817","heartbeat_back_addr":"172.21.15.161:6813/40817","heartbeat_front_addr":"172.21.15.161:6808/40817","state":["exists","up"]},{"osd":1,"uuid":"1d2b9892-a981-4801-bb85-7f2006509d26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":40831},{"type":"v1","addr":"172.21.15.161:6817","nonce":40831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":40831},{"type":"v1","addr":"172.21.15.161:6819","nonce":40831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":40831},{"type":"v1","addr":"172.21.15.161:6823","nonce":40831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":40831},{"type":"v1","addr":"172.21.15.161:6821","nonce":40831}]},"public_addr":"172.21.15.161:6817/40831","cluster_addr":"172.21.15.161:6819/40831","heartbeat_back_addr":"172.21.15.161:6823/40831","heartbeat_front_addr":"172.21.15.161:6821/40831","state":["exists","up"]},{"osd":2,"uuid":"9d0ae2f2-18b8-4349-abba-bfa054f968ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":40818},{"type":"v1","addr":"172.21.15.161:6805","nonce":40818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6807","nonce":40818},{"type":"v1","addr":"172.21.15.161:6809","nonce":40818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":40818},{"type":"v1","addr":"172.21.15.161:6815","nonce":40818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6811","nonce":40818},{"type":"v1","addr":"172.21.15.161:6812","nonce":40818}]},"public_addr":"172.21.15.161:6805/40818","cluster_addr":"172.21.15.161:6809/40818","heartbeat_back_addr":"172.21.15.161:6815/40818","heartbeat_front_addr":"172.21.15.161:6812/40818","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T04:28:22.726926+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T04:28:22.207840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T04:28:22.762138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T04:28:28.381 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-27T04:28:28.381 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T04:28:28.708 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:28.708 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":12,"fsid":"acaf0272-2637-4013-a083-8ec5702e3ecd","created":"2024-04-27T04:28:17.321378+0000","modified":"2024-04-27T04:28:28.556744+0000","last_up_change":"2024-04-27T04:28:25.094539+0000","last_in_change":"2024-04-27T04:28:18.306825+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T04:28:25.592985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-27T04:28:26.125142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c03baab-c9f4-452c-9c03-407aefaa7556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":40817},{"type":"v1","addr":"172.21.15.161:6801","nonce":40817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":40817},{"type":"v1","addr":"172.21.15.161:6803","nonce":40817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":40817},{"type":"v1","addr":"172.21.15.161:6813","nonce":40817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":40817},{"type":"v1","addr":"172.21.15.161:6808","nonce":40817}]},"public_addr":"172.21.15.161:6801/40817","cluster_addr":"172.21.15.161:6803/40817","heartbeat_back_addr":"172.21.15.161:6813/40817","heartbeat_front_addr":"172.21.15.161:6808/40817","state":["exists","up"]},{"osd":1,"uuid":"1d2b9892-a981-4801-bb85-7f2006509d26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":40831},{"type":"v1","addr":"172.21.15.161:6817","nonce":40831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":40831},{"type":"v1","addr":"172.21.15.161:6819","nonce":40831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":40831},{"type":"v1","addr":"172.21.15.161:6823","nonce":40831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":40831},{"type":"v1","addr":"172.21.15.161:6821","nonce":40831}]},"public_addr":"172.21.15.161:6817/40831","cluster_addr":"172.21.15.161:6819/40831","heartbeat_back_addr":"172.21.15.161:6823/40831","heartbeat_front_addr":"172.21.15.161:6821/40831","state":["exists","up"]},{"osd":2,"uuid":"9d0ae2f2-18b8-4349-abba-bfa054f968ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":40818},{"type":"v1","addr":"172.21.15.161:6805","nonce":40818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6807","nonce":40818},{"type":"v1","addr":"172.21.15.161:6809","nonce":40818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":40818},{"type":"v1","addr":"172.21.15.161:6815","nonce":40818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6811","nonce":40818},{"type":"v1","addr":"172.21.15.161:6812","nonce":40818}]},"public_addr":"172.21.15.161:6805/40818","cluster_addr":"172.21.15.161:6809/40818","heartbeat_back_addr":"172.21.15.161:6815/40818","heartbeat_front_addr":"172.21.15.161:6812/40818","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T04:28:22.726926+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T04:28:22.207840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T04:28:22.762138+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T04:28:28.722 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-27T04:28:28.723 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-27T04:28:28.723 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-27T04:28:28.890 INFO:teuthology.orchestra.run.smithi161.stdout:34359738370 2024-04-27T04:28:28.890 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-27T04:28:28.897 INFO:teuthology.orchestra.run.smithi161.stdout:38654705667 2024-04-27T04:28:28.898 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-27T04:28:28.899 INFO:teuthology.orchestra.run.smithi161.stdout:38654705666 2024-04-27T04:28:28.899 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-27T04:28:29.247 INFO:teuthology.orchestra.run.smithi161.stdout:34359738369 2024-04-27T04:28:29.261 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.1 2024-04-27T04:28:29.307 INFO:teuthology.orchestra.run.smithi161.stdout:38654705665 2024-04-27T04:28:29.319 INFO:teuthology.orchestra.run.smithi161.stdout:38654705666 2024-04-27T04:28:29.322 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.0 2024-04-27T04:28:29.334 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.2 2024-04-27T04:28:30.262 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-27T04:28:30.323 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-27T04:28:30.334 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-27T04:28:30.622 INFO:teuthology.orchestra.run.smithi161.stdout:34359738370 2024-04-27T04:28:30.642 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.1 2024-04-27T04:28:30.642 DEBUG:teuthology.parallel:result is None 2024-04-27T04:28:30.677 INFO:teuthology.orchestra.run.smithi161.stdout:38654705666 2024-04-27T04:28:30.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.0 2024-04-27T04:28:30.692 DEBUG:teuthology.parallel:result is None 2024-04-27T04:28:30.727 INFO:teuthology.orchestra.run.smithi161.stdout:38654705667 2024-04-27T04:28:30.741 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.2 2024-04-27T04:28:30.741 DEBUG:teuthology.parallel:result is None 2024-04-27T04:28:30.741 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-27T04:28:30.742 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-27T04:28:31.103 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:31.105 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-04-27T04:28:31.116 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-27T04:28:29.552276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82640,"kb_used_data":1456,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490160,"statfs":{"total":322122547200,"available":322037923840,"internally_reserved":0,"allocated":1490944,"data_stored":1259332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:27.566584+0000","last_change":"2024-04-27T04:28:27.564427+0000","last_active":"2024-04-27T04:28:27.566584+0000","last_peered":"2024-04-27T04:28:27.566584+0000","last_clean":"2024-04-27T04:28:27.566584+0000","last_became_active":"2024-04-27T04:28:27.564142+0000","last_became_peered":"2024-04-27T04:28:27.564142+0000","last_unstale":"2024-04-27T04:28:27.566584+0000","last_undegraded":"2024-04-27T04:28:27.566584+0000","last_fullsized":"2024-04-27T04:28:27.566584+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T16:14:13.521618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:27.566496+0000","last_change":"2024-04-27T04:28:27.564813+0000","last_active":"2024-04-27T04:28:27.566496+0000","last_peered":"2024-04-27T04:28:27.566496+0000","last_clean":"2024-04-27T04:28:27.566496+0000","last_became_active":"2024-04-27T04:28:27.563730+0000","last_became_peered":"2024-04-27T04:28:27.563730+0000","last_unstale":"2024-04-27T04:28:27.566496+0000","last_undegraded":"2024-04-27T04:28:27.566496+0000","last_fullsized":"2024-04-27T04:28:27.566496+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:54:45.252129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:27.566160+0000","last_change":"2024-04-27T04:28:27.563320+0000","last_active":"2024-04-27T04:28:27.566160+0000","last_peered":"2024-04-27T04:28:27.566160+0000","last_clean":"2024-04-27T04:28:27.566160+0000","last_became_active":"2024-04-27T04:28:27.563145+0000","last_became_peered":"2024-04-27T04:28:27.563145+0000","last_unstale":"2024-04-27T04:28:27.566160+0000","last_undegraded":"2024-04-27T04:28:27.566160+0000","last_fullsized":"2024-04-27T04:28:27.566160+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:27:28.213713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-27T04:28:28.566387+0000","last_change":"2024-04-27T04:28:27.567170+0000","last_active":"2024-04-27T04:28:28.566387+0000","last_peered":"2024-04-27T04:28:28.566387+0000","last_clean":"2024-04-27T04:28:28.566387+0000","last_became_active":"2024-04-27T04:28:27.566855+0000","last_became_peered":"2024-04-27T04:28:27.566855+0000","last_unstale":"2024-04-27T04:28:28.566387+0000","last_undegraded":"2024-04-27T04:28:28.566387+0000","last_fullsized":"2024-04-27T04:28:28.566387+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:57.220792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:28.041372+0000","last_change":"2024-04-27T04:28:27.563414+0000","last_active":"2024-04-27T04:28:28.041372+0000","last_peered":"2024-04-27T04:28:28.041372+0000","last_clean":"2024-04-27T04:28:28.041372+0000","last_became_active":"2024-04-27T04:28:27.563146+0000","last_became_peered":"2024-04-27T04:28:27.563146+0000","last_unstale":"2024-04-27T04:28:28.041372+0000","last_undegraded":"2024-04-27T04:28:28.041372+0000","last_fullsized":"2024-04-27T04:28:28.041372+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T16:12:43.281351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:27.566419+0000","last_change":"2024-04-27T04:28:27.564245+0000","last_active":"2024-04-27T04:28:27.566419+0000","last_peered":"2024-04-27T04:28:27.566419+0000","last_clean":"2024-04-27T04:28:27.566419+0000","last_became_active":"2024-04-27T04:28:27.563942+0000","last_became_peered":"2024-04-27T04:28:27.563942+0000","last_unstale":"2024-04-27T04:28:27.566419+0000","last_undegraded":"2024-04-27T04:28:27.566419+0000","last_fullsized":"2024-04-27T04:28:27.566419+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:43:38.852865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-27T04:28:28.566387+0000","last_change":"2024-04-27T04:28:27.567152+0000","last_active":"2024-04-27T04:28:28.566387+0000","last_peered":"2024-04-27T04:28:28.566387+0000","last_clean":"2024-04-27T04:28:28.566387+0000","last_became_active":"2024-04-27T04:28:27.566870+0000","last_became_peered":"2024-04-27T04:28:27.566870+0000","last_unstale":"2024-04-27T04:28:28.566387+0000","last_undegraded":"2024-04-27T04:28:28.566387+0000","last_fullsized":"2024-04-27T04:28:28.566387+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:42:16.797853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:27.566023+0000","last_change":"2024-04-27T04:28:27.564711+0000","last_active":"2024-04-27T04:28:27.566023+0000","last_peered":"2024-04-27T04:28:27.566023+0000","last_clean":"2024-04-27T04:28:27.566023+0000","last_became_active":"2024-04-27T04:28:27.564475+0000","last_became_peered":"2024-04-27T04:28:27.564475+0000","last_unstale":"2024-04-27T04:28:27.566023+0000","last_undegraded":"2024-04-27T04:28:27.566023+0000","last_fullsized":"2024-04-27T04:28:27.566023+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:15.740602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:27.566555+0000","last_change":"2024-04-27T04:28:27.564523+0000","last_active":"2024-04-27T04:28:27.566555+0000","last_peered":"2024-04-27T04:28:27.566555+0000","last_clean":"2024-04-27T04:28:27.566555+0000","last_became_active":"2024-04-27T04:28:27.563865+0000","last_became_peered":"2024-04-27T04:28:27.563865+0000","last_unstale":"2024-04-27T04:28:27.566555+0000","last_undegraded":"2024-04-27T04:28:27.566555+0000","last_fullsized":"2024-04-27T04:28:27.566555+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:32:19.765726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":106496,"data_stored":28764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27820,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T04:28:31.117 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-27T04:28:31.442 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:31.444 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-04-27T04:28:31.456 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-27T04:28:29.552276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82640,"kb_used_data":1456,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490160,"statfs":{"total":322122547200,"available":322037923840,"internally_reserved":0,"allocated":1490944,"data_stored":1259332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:27.566584+0000","last_change":"2024-04-27T04:28:27.564427+0000","last_active":"2024-04-27T04:28:27.566584+0000","last_peered":"2024-04-27T04:28:27.566584+0000","last_clean":"2024-04-27T04:28:27.566584+0000","last_became_active":"2024-04-27T04:28:27.564142+0000","last_became_peered":"2024-04-27T04:28:27.564142+0000","last_unstale":"2024-04-27T04:28:27.566584+0000","last_undegraded":"2024-04-27T04:28:27.566584+0000","last_fullsized":"2024-04-27T04:28:27.566584+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T16:14:13.521618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:27.566496+0000","last_change":"2024-04-27T04:28:27.564813+0000","last_active":"2024-04-27T04:28:27.566496+0000","last_peered":"2024-04-27T04:28:27.566496+0000","last_clean":"2024-04-27T04:28:27.566496+0000","last_became_active":"2024-04-27T04:28:27.563730+0000","last_became_peered":"2024-04-27T04:28:27.563730+0000","last_unstale":"2024-04-27T04:28:27.566496+0000","last_undegraded":"2024-04-27T04:28:27.566496+0000","last_fullsized":"2024-04-27T04:28:27.566496+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:54:45.252129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:27.566160+0000","last_change":"2024-04-27T04:28:27.563320+0000","last_active":"2024-04-27T04:28:27.566160+0000","last_peered":"2024-04-27T04:28:27.566160+0000","last_clean":"2024-04-27T04:28:27.566160+0000","last_became_active":"2024-04-27T04:28:27.563145+0000","last_became_peered":"2024-04-27T04:28:27.563145+0000","last_unstale":"2024-04-27T04:28:27.566160+0000","last_undegraded":"2024-04-27T04:28:27.566160+0000","last_fullsized":"2024-04-27T04:28:27.566160+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:27:28.213713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-27T04:28:28.566387+0000","last_change":"2024-04-27T04:28:27.567170+0000","last_active":"2024-04-27T04:28:28.566387+0000","last_peered":"2024-04-27T04:28:28.566387+0000","last_clean":"2024-04-27T04:28:28.566387+0000","last_became_active":"2024-04-27T04:28:27.566855+0000","last_became_peered":"2024-04-27T04:28:27.566855+0000","last_unstale":"2024-04-27T04:28:28.566387+0000","last_undegraded":"2024-04-27T04:28:28.566387+0000","last_fullsized":"2024-04-27T04:28:28.566387+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:57.220792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:28.041372+0000","last_change":"2024-04-27T04:28:27.563414+0000","last_active":"2024-04-27T04:28:28.041372+0000","last_peered":"2024-04-27T04:28:28.041372+0000","last_clean":"2024-04-27T04:28:28.041372+0000","last_became_active":"2024-04-27T04:28:27.563146+0000","last_became_peered":"2024-04-27T04:28:27.563146+0000","last_unstale":"2024-04-27T04:28:28.041372+0000","last_undegraded":"2024-04-27T04:28:28.041372+0000","last_fullsized":"2024-04-27T04:28:28.041372+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T16:12:43.281351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:27.566419+0000","last_change":"2024-04-27T04:28:27.564245+0000","last_active":"2024-04-27T04:28:27.566419+0000","last_peered":"2024-04-27T04:28:27.566419+0000","last_clean":"2024-04-27T04:28:27.566419+0000","last_became_active":"2024-04-27T04:28:27.563942+0000","last_became_peered":"2024-04-27T04:28:27.563942+0000","last_unstale":"2024-04-27T04:28:27.566419+0000","last_undegraded":"2024-04-27T04:28:27.566419+0000","last_fullsized":"2024-04-27T04:28:27.566419+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:43:38.852865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-27T04:28:28.566387+0000","last_change":"2024-04-27T04:28:27.567152+0000","last_active":"2024-04-27T04:28:28.566387+0000","last_peered":"2024-04-27T04:28:28.566387+0000","last_clean":"2024-04-27T04:28:28.566387+0000","last_became_active":"2024-04-27T04:28:27.566870+0000","last_became_peered":"2024-04-27T04:28:27.566870+0000","last_unstale":"2024-04-27T04:28:28.566387+0000","last_undegraded":"2024-04-27T04:28:28.566387+0000","last_fullsized":"2024-04-27T04:28:28.566387+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:42:16.797853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:27.566023+0000","last_change":"2024-04-27T04:28:27.564711+0000","last_active":"2024-04-27T04:28:27.566023+0000","last_peered":"2024-04-27T04:28:27.566023+0000","last_clean":"2024-04-27T04:28:27.566023+0000","last_became_active":"2024-04-27T04:28:27.564475+0000","last_became_peered":"2024-04-27T04:28:27.564475+0000","last_unstale":"2024-04-27T04:28:27.566023+0000","last_undegraded":"2024-04-27T04:28:27.566023+0000","last_fullsized":"2024-04-27T04:28:27.566023+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:15.740602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T04:28:27.566555+0000","last_change":"2024-04-27T04:28:27.564523+0000","last_active":"2024-04-27T04:28:27.566555+0000","last_peered":"2024-04-27T04:28:27.566555+0000","last_clean":"2024-04-27T04:28:27.566555+0000","last_became_active":"2024-04-27T04:28:27.563865+0000","last_became_peered":"2024-04-27T04:28:27.563865+0000","last_unstale":"2024-04-27T04:28:27.566555+0000","last_undegraded":"2024-04-27T04:28:27.566555+0000","last_fullsized":"2024-04-27T04:28:27.566555+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T04:28:26.548977+0000","last_clean_scrub_stamp":"2024-04-27T04:28:26.548977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:32:19.765726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":106496,"data_stored":28764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27820,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T04:28:31.456 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-27T04:28:31.456 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-27T04:28:31.457 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-27T04:28:31.457 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-27T04:28:31.824 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:31.825 INFO:teuthology.orchestra.run.smithi161.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-27T04:28:31.838 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-27T04:28:31.839 INFO:teuthology.run_tasks:Running task rgw... 2024-04-27T04:28:31.852 DEBUG:tasks.rgw:config is {'client.0': None} 2024-04-27T04:28:31.852 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2024-04-27T04:28:31.852 INFO:tasks.rgw:Creating data pools 2024-04-27T04:28:31.852 DEBUG:tasks.rgw:Obtaining remote for client client.0 2024-04-27T04:28:31.852 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2024-04-27T04:28:32.581 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'default.rgw.buckets.data' created 2024-04-27T04:28:32.597 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2024-04-27T04:28:33.598 INFO:teuthology.orchestra.run.smithi161.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-27T04:28:33.615 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2024-04-27T04:28:34.589 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'default.rgw.buckets.index' created 2024-04-27T04:28:34.608 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2024-04-27T04:28:35.613 INFO:teuthology.orchestra.run.smithi161.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-27T04:28:35.630 DEBUG:tasks.rgw:Pools created 2024-04-27T04:28:35.630 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2024-04-27T04:28:35.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2024-04-27T04:28:35.630 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2024-04-27T04:28:35.703 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:35.703 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:35.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:35.712+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.619+0000 7efcacd26780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:37.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.619+0000 7efcacd26780 20 realm 2024-04-27T04:28:37.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.619+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.619+0000 7efcacd26780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:37.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.619+0000 7efcacd26780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:28:37.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.619+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.620+0000 7efcacd26780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:37.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.620+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.620+0000 7efcacd26780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:37.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.620+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.620+0000 7efcacd26780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:37.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.620+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.621+0000 7efcacd26780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:37.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.621+0000 7efcacd26780 10 Creating default zonegroup 2024-04-27T04:28:37.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.621+0000 7efcacd26780 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-04-27T04:28:37.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.631+0000 7efcacd26780 10 WARNING: driver->list_zones() returned r=-2 2024-04-27T04:28:37.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.631+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.631+0000 7efcacd26780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:37.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.635+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.635+0000 7efcacd26780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:37.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.635+0000 7efcacd26780 10 could not read realm id: (2) No such file or directory 2024-04-27T04:28:37.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.635+0000 7efcacd26780 10 WARNING: failed to set zone as default, r=-22 2024-04-27T04:28:37.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.635+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.635+0000 7efcacd26780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:37.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.641+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.641+0000 7efcacd26780 20 rados_obj.operate() r=0 bl.length=874 2024-04-27T04:28:37.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.641+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 20 rados_obj.operate() r=0 bl.length=420 2024-04-27T04:28:37.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 10 Using default name default 2024-04-27T04:28:37.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:37.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 20 rados_obj.operate() r=0 bl.length=874 2024-04-27T04:28:37.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 20 zone default found 2024-04-27T04:28:37.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 4 Realm: () 2024-04-27T04:28:37.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:28:37.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:28:37.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:28:37.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 20 zonegroup default 2024-04-27T04:28:37.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.642+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:37.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.643+0000 7efcacd26780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:37.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:37.643+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:39.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:39.627+0000 7efcacd26780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:39.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:39.627+0000 7efcacd26780 20 rados->read ofs=0 len=0 2024-04-27T04:28:39.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:39.628+0000 7efcacd26780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:39.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:39.628+0000 7efcacd26780 20 started sync module instance, tier type = 2024-04-27T04:28:39.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:39.628+0000 7efcacd26780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:28:41.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.677+0000 7efcacd26780 20 add_watcher() i=0 2024-04-27T04:28:41.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.677+0000 7efcacd26780 20 add_watcher() i=1 2024-04-27T04:28:41.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.677+0000 7efcacd26780 20 add_watcher() i=2 2024-04-27T04:28:41.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.677+0000 7efcacd26780 20 add_watcher() i=3 2024-04-27T04:28:41.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.677+0000 7efcacd26780 20 add_watcher() i=4 2024-04-27T04:28:41.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.677+0000 7efcacd26780 20 add_watcher() i=5 2024-04-27T04:28:41.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.677+0000 7efcacd26780 20 add_watcher() i=6 2024-04-27T04:28:41.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.680+0000 7efcacd26780 20 add_watcher() i=7 2024-04-27T04:28:41.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.680+0000 7efcacd26780 2 all 8 watchers are set, enabling cache 2024-04-27T04:28:41.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.680+0000 7efcacd26780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-27T04:28:41.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.680+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-27T04:28:41.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.680+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.680+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-27T04:28:41.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.680+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-27T04:28:41.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.680+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-27T04:28:41.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-27T04:28:41.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-27T04:28:41.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-27T04:28:41.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-27T04:28:41.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.4 2024-04-27T04:28:41.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.4 does not exist 2024-04-27T04:28:41.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.5 2024-04-27T04:28:41.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.5 does not exist 2024-04-27T04:28:41.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.6 2024-04-27T04:28:41.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.681+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.6 does not exist 2024-04-27T04:28:41.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.7 2024-04-27T04:28:41.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.7 does not exist 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.8 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.8 does not exist 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.9 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.9 does not exist 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.10 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.10 does not exist 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.11 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.682+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.683+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.11 does not exist 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.683+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.12 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.683+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.683+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.12 does not exist 2024-04-27T04:28:41.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.683+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.13 2024-04-27T04:28:41.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.683+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.683+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.13 does not exist 2024-04-27T04:28:41.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.683+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.14 2024-04-27T04:28:41.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.683+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.683+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.14 does not exist 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.683+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.15 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.683+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.15 does not exist 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.16 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.16 does not exist 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.17 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.17 does not exist 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.18 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.18 does not exist 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.19 2024-04-27T04:28:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.19 does not exist 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.20 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.684+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.20 does not exist 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.21 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.21 does not exist 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.22 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.22 does not exist 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.23 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.23 does not exist 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.24 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.24 does not exist 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.25 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.685+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.25 does not exist 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.26 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.26 does not exist 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.27 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.27 does not exist 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.28 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.28 does not exist 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.29 2024-04-27T04:28:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.29 does not exist 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.30 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.686+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.30 does not exist 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.31 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.31 does not exist 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.32 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.32 does not exist 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.33 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.33 does not exist 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.34 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.34 does not exist 2024-04-27T04:28:41.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.35 2024-04-27T04:28:41.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.687+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.35 does not exist 2024-04-27T04:28:41.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.36 2024-04-27T04:28:41.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.36 does not exist 2024-04-27T04:28:41.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.37 2024-04-27T04:28:41.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.37 does not exist 2024-04-27T04:28:41.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.38 2024-04-27T04:28:41.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.38 does not exist 2024-04-27T04:28:41.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.39 2024-04-27T04:28:41.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.39 does not exist 2024-04-27T04:28:41.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.40 2024-04-27T04:28:41.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.688+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.40 does not exist 2024-04-27T04:28:41.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.41 2024-04-27T04:28:41.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.41 does not exist 2024-04-27T04:28:41.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.42 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.42 does not exist 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.43 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.43 does not exist 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.44 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.44 does not exist 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.45 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.689+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.45 does not exist 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.46 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.46 does not exist 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.47 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.47 does not exist 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.48 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.48 does not exist 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.49 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.49 does not exist 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.50 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.690+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.50 does not exist 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.51 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.51 does not exist 2024-04-27T04:28:41.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.52 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.52 does not exist 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.53 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.53 does not exist 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.54 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.54 does not exist 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.55 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.691+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.55 does not exist 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.56 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.56 does not exist 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.57 2024-04-27T04:28:41.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.57 does not exist 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.58 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.58 does not exist 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.59 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.59 does not exist 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.60 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.692+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.60 does not exist 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.61 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.61 does not exist 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.62 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.62 does not exist 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.63 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.63 does not exist 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.64 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.64 does not exist 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.65 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.693+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.65 does not exist 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.66 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.66 does not exist 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.67 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.67 does not exist 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.68 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.68 does not exist 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.69 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.69 does not exist 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.70 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.694+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.70 does not exist 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.71 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.71 does not exist 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.72 2024-04-27T04:28:41.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.72 does not exist 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.73 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.73 does not exist 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.74 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.74 does not exist 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.75 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.695+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.75 does not exist 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.76 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.76 does not exist 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.77 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.77 does not exist 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.78 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.78 does not exist 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.79 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.79 does not exist 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.80 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.696+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.80 does not exist 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.81 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.81 does not exist 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.82 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.82 does not exist 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.83 2024-04-27T04:28:41.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.83 does not exist 2024-04-27T04:28:41.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.84 2024-04-27T04:28:41.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.84 does not exist 2024-04-27T04:28:41.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.85 2024-04-27T04:28:41.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.697+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.85 does not exist 2024-04-27T04:28:41.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.86 2024-04-27T04:28:41.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.86 does not exist 2024-04-27T04:28:41.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.87 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.87 does not exist 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.88 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.88 does not exist 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.89 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.89 does not exist 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.90 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.698+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.90 does not exist 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.91 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.91 does not exist 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.92 2024-04-27T04:28:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.92 does not exist 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.93 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.93 does not exist 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.94 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.94 does not exist 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.95 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.95 does not exist 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.96 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.699+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.96 does not exist 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.97 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.97 does not exist 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.98 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.98 does not exist 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.99 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.99 does not exist 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.100 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.100 does not exist 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.101 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.700+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.701+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.101 does not exist 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.701+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.102 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.701+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.701+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.102 does not exist 2024-04-27T04:28:41.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.701+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.103 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.701+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.701+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.103 does not exist 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.701+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.104 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.701+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.701+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.104 does not exist 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.701+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.105 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.701+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.105 does not exist 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.106 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.106 does not exist 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.107 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.107 does not exist 2024-04-27T04:28:41.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.108 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.108 does not exist 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.109 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.109 does not exist 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.110 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.702+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.110 does not exist 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.111 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.111 does not exist 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.112 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.112 does not exist 2024-04-27T04:28:41.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.113 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.113 does not exist 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.114 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.114 does not exist 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.115 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.115 does not exist 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.116 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.703+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.704+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.116 does not exist 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.704+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.117 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.704+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.704+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.117 does not exist 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.704+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.118 2024-04-27T04:28:41.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.704+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.704+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.118 does not exist 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.704+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.119 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.704+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.704+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.119 does not exist 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.704+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.120 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.704+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.120 does not exist 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.121 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.121 does not exist 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.122 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.122 does not exist 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.123 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.123 does not exist 2024-04-27T04:28:41.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.124 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.124 does not exist 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.125 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.705+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.125 does not exist 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.706+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.126 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.706+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.706+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.126 does not exist 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.706+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.127 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.706+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.706+0000 7efcacd26780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.127 does not exist 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.706+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-04-27T04:28:41.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.707+0000 7efcacd26780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-27T04:28:41.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.713+0000 7efc91ca1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:41.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.713+0000 7efcacd26780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:41.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:41.713+0000 7efcacd26780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:44.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.635+0000 7efcacd26780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-27T04:28:44.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.635+0000 7efcacd26780 5 note: GC not initialized 2024-04-27T04:28:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.636+0000 7efc8d498700 20 reqs_thread_entry: start 2024-04-27T04:28:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.636+0000 7efcacd26780 20 init_complete bucket index max shards: 11 2024-04-27T04:28:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.636+0000 7efcacd26780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:28:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.636+0000 7efc8b494700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:29:15 2024 2024-04-27T04:28:44.637 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:28:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.636+0000 7efc8ac72700 20 reqs_thread_entry: start 2024-04-27T04:28:44.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.655+0000 7efc91ca1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:44.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.659+0000 7efcacd26780 20 remove_watcher() i=0 2024-04-27T04:28:44.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.659+0000 7efcacd26780 2 removed watcher, disabling cache 2024-04-27T04:28:44.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.660+0000 7efcacd26780 20 remove_watcher() i=1 2024-04-27T04:28:44.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.663+0000 7efcacd26780 20 remove_watcher() i=2 2024-04-27T04:28:44.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.665+0000 7efcacd26780 20 remove_watcher() i=3 2024-04-27T04:28:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.667+0000 7efcacd26780 20 remove_watcher() i=4 2024-04-27T04:28:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.670+0000 7efcacd26780 20 remove_watcher() i=5 2024-04-27T04:28:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.671+0000 7efcacd26780 20 remove_watcher() i=6 2024-04-27T04:28:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.672+0000 7efcacd26780 20 remove_watcher() i=7 2024-04-27T04:28:44.675 INFO:teuthology.orchestra.run.smithi161.stdout:[] 2024-04-27T04:28:44.677 DEBUG:tasks.util.rgw: json result: [] 2024-04-27T04:28:44.677 INFO:tasks.rgw:Configuring storage class type = LUKEWARM 2024-04-27T04:28:44.677 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2024-04-27T04:28:44.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2024-04-27T04:28:44.678 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM 2024-04-27T04:28:44.730 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:44.750 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:44.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.739+0000 7f78801c2780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:44.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.739+0000 7f78801c2780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:44.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.743+0000 7f7861936700 20 reqs_thread_entry: start 2024-04-27T04:28:44.751 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-04-27T04:28:44.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:44.752 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "default-placement", 2024-04-27T04:28:44.752 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2024-04-27T04:28:44.752 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "default-placement", 2024-04-27T04:28:44.752 INFO:teuthology.orchestra.run.smithi161.stdout: "tags": [], 2024-04-27T04:28:44.752 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": [ 2024-04-27T04:28:44.752 INFO:teuthology.orchestra.run.smithi161.stdout: "LUKEWARM", 2024-04-27T04:28:44.752 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD" 2024-04-27T04:28:44.752 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-04-27T04:28:44.752 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:44.752 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:44.752 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-04-27T04:28:44.753 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['LUKEWARM', 'STANDARD']}}] 2024-04-27T04:28:44.753 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2024-04-27T04:28:44.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2024-04-27T04:28:44.754 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2024-04-27T04:28:44.804 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:44.804 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:44.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.813+0000 7fecec739780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:44.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.813+0000 7fecec739780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:44.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.816+0000 7feccdead700 20 reqs_thread_entry: start 2024-04-27T04:28:44.822 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-04-27T04:28:44.822 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "93e3c4d1-2922-476e-bc51-fb4366c74703", 2024-04-27T04:28:44.822 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "default", 2024-04-27T04:28:44.822 INFO:teuthology.orchestra.run.smithi161.stdout: "domain_root": "default.rgw.meta:root", 2024-04-27T04:28:44.822 INFO:teuthology.orchestra.run.smithi161.stdout: "control_pool": "default.rgw.control", 2024-04-27T04:28:44.822 INFO:teuthology.orchestra.run.smithi161.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-27T04:28:44.822 INFO:teuthology.orchestra.run.smithi161.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-27T04:28:44.822 INFO:teuthology.orchestra.run.smithi161.stdout: "log_pool": "default.rgw.log", 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: "otp_pool": "default.rgw.otp", 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: "system_key": { 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "", 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "" 2024-04-27T04:28:44.823 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_pools": [ 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "default-placement", 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": { 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: "LUKEWARM": { 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: "data_pool": "default.rgw.buckets.data.lukewarm" 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD": { 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: "data_pool": "default.rgw.buckets.data" 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:44.824 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:44.825 INFO:teuthology.orchestra.run.smithi161.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-27T04:28:44.825 INFO:teuthology.orchestra.run.smithi161.stdout: "index_type": 0, 2024-04-27T04:28:44.825 INFO:teuthology.orchestra.run.smithi161.stdout: "inline_data": true 2024-04-27T04:28:44.825 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:44.825 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:44.825 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:44.825 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "", 2024-04-27T04:28:44.825 INFO:teuthology.orchestra.run.smithi161.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-27T04:28:44.825 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-04-27T04:28:44.826 DEBUG:tasks.util.rgw: json result: {'id': '93e3c4d1-2922-476e-bc51-fb4366c74703', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-27T04:28:44.826 INFO:tasks.rgw:Configuring storage class type = FROZEN 2024-04-27T04:28:44.826 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2024-04-27T04:28:44.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2024-04-27T04:28:44.826 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN 2024-04-27T04:28:44.874 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:44.874 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:44.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.883+0000 7f8626497780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:44.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.883+0000 7f8626497780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:44.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.886+0000 7f8607c0b700 20 reqs_thread_entry: start 2024-04-27T04:28:44.892 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-04-27T04:28:44.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:44.892 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "default-placement", 2024-04-27T04:28:44.892 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2024-04-27T04:28:44.892 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "default-placement", 2024-04-27T04:28:44.892 INFO:teuthology.orchestra.run.smithi161.stdout: "tags": [], 2024-04-27T04:28:44.892 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": [ 2024-04-27T04:28:44.892 INFO:teuthology.orchestra.run.smithi161.stdout: "FROZEN", 2024-04-27T04:28:44.892 INFO:teuthology.orchestra.run.smithi161.stdout: "LUKEWARM", 2024-04-27T04:28:44.892 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD" 2024-04-27T04:28:44.892 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-04-27T04:28:44.893 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:44.893 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:44.893 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-04-27T04:28:44.893 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2024-04-27T04:28:44.894 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2024-04-27T04:28:44.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2024-04-27T04:28:44.894 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2024-04-27T04:28:44.943 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:44.944 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:44.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.952+0000 7fa6ad1c7780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:44.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.952+0000 7fa6ad1c7780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:44.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:44.955+0000 7fa68e93b700 20 reqs_thread_entry: start 2024-04-27T04:28:44.961 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-04-27T04:28:44.961 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "93e3c4d1-2922-476e-bc51-fb4366c74703", 2024-04-27T04:28:44.961 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "default", 2024-04-27T04:28:44.961 INFO:teuthology.orchestra.run.smithi161.stdout: "domain_root": "default.rgw.meta:root", 2024-04-27T04:28:44.961 INFO:teuthology.orchestra.run.smithi161.stdout: "control_pool": "default.rgw.control", 2024-04-27T04:28:44.961 INFO:teuthology.orchestra.run.smithi161.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-27T04:28:44.962 INFO:teuthology.orchestra.run.smithi161.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-27T04:28:44.962 INFO:teuthology.orchestra.run.smithi161.stdout: "log_pool": "default.rgw.log", 2024-04-27T04:28:44.962 INFO:teuthology.orchestra.run.smithi161.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-27T04:28:44.962 INFO:teuthology.orchestra.run.smithi161.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-27T04:28:44.962 INFO:teuthology.orchestra.run.smithi161.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-27T04:28:44.962 INFO:teuthology.orchestra.run.smithi161.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-27T04:28:44.962 INFO:teuthology.orchestra.run.smithi161.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-27T04:28:44.962 INFO:teuthology.orchestra.run.smithi161.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-27T04:28:44.962 INFO:teuthology.orchestra.run.smithi161.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-27T04:28:44.962 INFO:teuthology.orchestra.run.smithi161.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-27T04:28:44.962 INFO:teuthology.orchestra.run.smithi161.stdout: "otp_pool": "default.rgw.otp", 2024-04-27T04:28:44.962 INFO:teuthology.orchestra.run.smithi161.stdout: "system_key": { 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "", 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "" 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_pools": [ 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "default-placement", 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": { 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: "FROZEN": { 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: "data_pool": "default.rgw.buckets.data.frozen" 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: "LUKEWARM": { 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: "data_pool": "default.rgw.buckets.data.lukewarm" 2024-04-27T04:28:44.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD": { 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout: "data_pool": "default.rgw.buckets.data" 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout: "index_type": 0, 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout: "inline_data": true 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "", 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-27T04:28:44.964 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-04-27T04:28:44.965 DEBUG:tasks.util.rgw: json result: {'id': '93e3c4d1-2922-476e-bc51-fb4366c74703', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-27T04:28:44.965 INFO:tasks.rgw:Starting rgw... 2024-04-27T04:28:44.965 INFO:tasks.rgw:rgw client.0 config is {} 2024-04-27T04:28:44.965 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-27T04:28:44.966 INFO:tasks.rgw.client.0:Restarting daemon 2024-04-27T04:28:44.966 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2024-04-27T04:28:44.968 INFO:tasks.rgw.client.0:Started 2024-04-27T04:28:44.968 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi161.front.sepia.ceph.com:80/ 2024-04-27T04:28:44.968 DEBUG:teuthology.orchestra.run.smithi161:> curl http://smithi161.front.sepia.ceph.com:80/ 2024-04-27T04:28:45.004 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-27T04:28:45.004 INFO:teuthology.orchestra.run.smithi161.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-27T04:28:45.004 INFO:teuthology.orchestra.run.smithi161.stderr: Dload Upload Total Spent Left Speed 2024-04-27T04:28:45.004 INFO:teuthology.orchestra.run.smithi161.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi161.front.sepia.ceph.com port 80: Connection refused 2024-04-27T04:28:46.005 DEBUG:teuthology.orchestra.run.smithi161:> curl http://smithi161.front.sepia.ceph.com:80/ 2024-04-27T04:28:46.027 INFO:teuthology.orchestra.run.smithi161.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-27T04:28:46.027 INFO:teuthology.orchestra.run.smithi161.stderr: Dload Upload Total Spent Left Speed 2024-04-27T04:28:46.027 INFO:teuthology.orchestra.run.smithi161.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-27T04:28:46.028 INFO:teuthology.orchestra.run.smithi161.stdout:anonymous 2024-04-27T04:28:46.028 INFO:teuthology.run_tasks:Running task tox... 2024-04-27T04:28:46.041 INFO:tasks.tox:Deploying tox from pip... 2024-04-27T04:28:46.041 DEBUG:teuthology.orchestra.run.smithi161:> python3 -m venv /home/ubuntu/cephtest/tox-venv 2024-04-27T04:28:47.562 DEBUG:teuthology.orchestra.run.smithi161:> source /home/ubuntu/cephtest/tox-venv/bin/activate && pip install tox==3.15.0 2024-04-27T04:28:47.819 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting tox==3.15.0 2024-04-27T04:28:48.151 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/37/70/14b3fac1447864facfea2b604d531d8ece2380fec3fbb6938f4b38ecb217/tox-3.15.0-py2.py3-none-any.whl (82kB) 2024-04-27T04:28:48.220 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting six<2,>=1.14.0 (from tox==3.15.0) 2024-04-27T04:28:48.323 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl 2024-04-27T04:28:48.342 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting filelock<4,>=3.0.0 (from tox==3.15.0) 2024-04-27T04:28:48.466 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/84/ce/8916d10ef537f3f3b046843255f9799504aa41862bfa87844b9bdc5361cd/filelock-3.4.1-py3-none-any.whl 2024-04-27T04:28:48.491 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting toml>=0.9.4 (from tox==3.15.0) 2024-04-27T04:28:48.587 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl 2024-04-27T04:28:48.608 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting py<2,>=1.4.17 (from tox==3.15.0) 2024-04-27T04:28:48.721 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl (98kB) 2024-04-27T04:28:48.756 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting packaging>=14 (from tox==3.15.0) 2024-04-27T04:28:48.875 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/05/8e/8de486cbd03baba4deef4142bd643a3e7bbe954a784dc1bb17142572d127/packaging-21.3-py3-none-any.whl (40kB) 2024-04-27T04:28:48.928 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting pluggy<1,>=0.12.0 (from tox==3.15.0) 2024-04-27T04:28:49.034 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/a0/28/85c7aa31b80d150b772fbe4a229487bc6644da9ccb7e427dd8cc60cb8a62/pluggy-0.13.1-py2.py3-none-any.whl 2024-04-27T04:28:49.058 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting importlib-metadata<2,>=0.12; python_version < "3.8" (from tox==3.15.0) 2024-04-27T04:28:49.228 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/8e/58/cdea07eb51fc2b906db0968a94700866fc46249bdc75cac23f9d13168929/importlib_metadata-1.7.0-py2.py3-none-any.whl 2024-04-27T04:28:49.259 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 (from tox==3.15.0) 2024-04-27T04:28:49.562 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/18/a2/7931d40ecb02b5236a34ac53770f2f6931e3082b7a7dafe915d892d749d6/virtualenv-20.17.1-py3-none-any.whl (8.8MB) 2024-04-27T04:28:50.236 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting pyparsing!=3.0.5,>=2.0.2 (from packaging>=14->tox==3.15.0) 2024-04-27T04:28:50.386 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/9d/ea/6d76df31432a0e6fdf81681a895f009a4bb47b3c39036db3e1b528191d52/pyparsing-3.1.2-py3-none-any.whl (103kB) 2024-04-27T04:28:50.415 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting zipp>=0.5 (from importlib-metadata<2,>=0.12; python_version < "3.8"->tox==3.15.0) 2024-04-27T04:28:50.536 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/bd/df/d4a4974a3e3957fd1c1fa3082366d7fff6e428ddb55f074bf64876f8e8ad/zipp-3.6.0-py3-none-any.whl 2024-04-27T04:28:50.564 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting importlib-resources>=5.4; python_version < "3.7" (from virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0->tox==3.15.0) 2024-04-27T04:28:50.699 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/24/1b/33e489669a94da3ef4562938cd306e8fa915e13939d7b8277cb5569cb405/importlib_resources-5.4.0-py3-none-any.whl 2024-04-27T04:28:50.732 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting platformdirs<3,>=2.4 (from virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0->tox==3.15.0) 2024-04-27T04:28:50.845 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/b1/78/dcfd84d3aabd46a9c77260fb47ea5d244806e4daef83aa6fe5d83adb182c/platformdirs-2.4.0-py3-none-any.whl 2024-04-27T04:28:50.871 INFO:teuthology.orchestra.run.smithi161.stdout:Collecting distlib<1,>=0.3.6 (from virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0->tox==3.15.0) 2024-04-27T04:28:50.970 INFO:teuthology.orchestra.run.smithi161.stdout: Downloading https://files.pythonhosted.org/packages/8e/41/9307e4f5f9976bc8b7fea0b66367734e8faf3ec84bc0d412d8cfabbb66cd/distlib-0.3.8-py2.py3-none-any.whl (468kB) 2024-04-27T04:28:51.017 INFO:teuthology.orchestra.run.smithi161.stdout:Installing collected packages: six, filelock, toml, py, pyparsing, packaging, zipp, importlib-metadata, pluggy, importlib-resources, platformdirs, distlib, virtualenv, tox 2024-04-27T04:28:51.432 INFO:teuthology.orchestra.run.smithi161.stdout:Successfully installed distlib-0.3.8 filelock-3.4.1 importlib-metadata-1.7.0 importlib-resources-5.4.0 packaging-21.3 platformdirs-2.4.0 pluggy-0.13.1 py-1.11.0 pyparsing-3.1.2 six-1.16.0 toml-0.10.2 tox-3.15.0 virtualenv-20.17.1 zipp-3.6.0 2024-04-27T04:28:51.631 INFO:teuthology.orchestra.run.smithi161.stderr:You are using pip version 9.0.3, however version 24.0 is available. 2024-04-27T04:28:51.631 INFO:teuthology.orchestra.run.smithi161.stderr:You should consider upgrading via the 'pip install --upgrade pip' command. 2024-04-27T04:28:51.667 INFO:teuthology.run_tasks:Running task s3tests... 2024-04-27T04:28:51.682 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'extra_attrs': ['lifecycle'], 'lc_debug_interval': 10, 'rgw_server': 'client.0', 'storage classes': 'LUKEWARM, FROZEN', 'force-branch': 'ceph-reef'}} 2024-04-27T04:28:51.682 INFO:tasks.s3tests:Downloading s3-tests... 2024-04-27T04:28:51.682 INFO:tasks.s3tests:Using branch 'ceph-reef' for s3tests 2024-04-27T04:28:51.683 DEBUG:teuthology.orchestra.run.smithi161:> git clone -b ceph-reef https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests-client.0 2024-04-27T04:28:51.699 INFO:teuthology.orchestra.run.smithi161.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests-client.0'... 2024-04-27T04:28:52.463 INFO:tasks.s3tests:Creating rgw users... 2024-04-27T04:28:52.464 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2024-04-27T04:28:52.464 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --email foo.client.0+test@test.test --caps 'user-policy=*' --access-key YQJFAXEUESMVBGWOCZSU --secret GRT8TQ4wLR9qJD3ReMBDYsuQc6GUzz0TfRzuH910LM09ZD3loGLmRw== --cluster ceph 2024-04-27T04:28:52.556 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:52.556 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:52.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.564+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.565+0000 7f2f8410e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.565+0000 7f2f8410e780 20 realm 2024-04-27T04:28:52.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.565+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.565+0000 7f2f8410e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.565+0000 7f2f8410e780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:28:52.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.565+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.565+0000 7f2f8410e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.565+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.566+0000 7f2f8410e780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:52.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.566+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.566+0000 7f2f8410e780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:28:52.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.566+0000 7f2f8410e780 20 searching for the correct realm 2024-04-27T04:28:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.573+0000 7f2f8410e780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.573+0000 7f2f8410e780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.573+0000 7f2f8410e780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.573+0000 7f2f8410e780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.573+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.573+0000 7f2f8410e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.573+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 zone default found 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 4 Realm: () 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:28:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:28:52.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:28:52.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 zonegroup default 2024-04-27T04:28:52.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 started sync module instance, tier type = 2024-04-27T04:28:52.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.574+0000 7f2f8410e780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:28:52.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.589+0000 7f2f8410e780 20 add_watcher() i=0 2024-04-27T04:28:52.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.589+0000 7f2f8410e780 20 add_watcher() i=1 2024-04-27T04:28:52.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.589+0000 7f2f8410e780 20 add_watcher() i=2 2024-04-27T04:28:52.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.589+0000 7f2f8410e780 20 add_watcher() i=3 2024-04-27T04:28:52.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.589+0000 7f2f8410e780 20 add_watcher() i=4 2024-04-27T04:28:52.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.589+0000 7f2f8410e780 20 add_watcher() i=5 2024-04-27T04:28:52.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.589+0000 7f2f8410e780 20 add_watcher() i=6 2024-04-27T04:28:52.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.589+0000 7f2f8410e780 20 add_watcher() i=7 2024-04-27T04:28:52.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.589+0000 7f2f8410e780 2 all 8 watchers are set, enabling cache 2024-04-27T04:28:52.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.591+0000 7f2f69089700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:52.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.591+0000 7f2f8410e780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:52.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.591+0000 7f2f8410e780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:52.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.595+0000 7f2f8410e780 5 note: GC not initialized 2024-04-27T04:28:52.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.595+0000 7f2f64880700 20 reqs_thread_entry: start 2024-04-27T04:28:52.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.595+0000 7f2f8410e780 20 init_complete bucket index max shards: 11 2024-04-27T04:28:52.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:28:52.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f6287c700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:29:23 2024 2024-04-27T04:28:52.596 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:28:52.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f6205a700 20 reqs_thread_entry: start 2024-04-27T04:28:52.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-04-27T04:28:52.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2024-04-27T04:28:52.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-04-27T04:28:52.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2024-04-27T04:28:52.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2024-04-27T04:28:52.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2024-04-27T04:28:52.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 10 cache get: name=default.rgw.meta+users.keys+YQJFAXEUESMVBGWOCZSU : miss 2024-04-27T04:28:52.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.596+0000 7f2f8410e780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.597+0000 7f2f8410e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.597+0000 7f2f8410e780 10 cache put: name=default.rgw.meta+users.keys+YQJFAXEUESMVBGWOCZSU info.flags=0x0 2024-04-27T04:28:52.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.597+0000 7f2f8410e780 10 adding default.rgw.meta+users.keys+YQJFAXEUESMVBGWOCZSU to cache LRU end 2024-04-27T04:28:52.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.597+0000 7f2f8410e780 10 cache get: name=default.rgw.meta+users.keys+YQJFAXEUESMVBGWOCZSU : hit (negative entry) 2024-04-27T04:28:52.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.597+0000 7f2f8410e780 10 cache get: name=default.rgw.meta+users.keys+YQJFAXEUESMVBGWOCZSU : hit (negative entry) 2024-04-27T04:28:52.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.598+0000 7f2f8410e780 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-04-27T04:28:52.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.598+0000 7f2f8410e780 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-04-27T04:28:52.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.598+0000 7f2f8410e780 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-04-27T04:28:52.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.598+0000 7f2f7c8b0700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 94346968201984 notifier 4250 bl.length()=640 2024-04-27T04:28:52.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.598+0000 7f2f7c8b0700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-04-27T04:28:52.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.598+0000 7f2f7c8b0700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-04-27T04:28:52.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.599+0000 7f2f8410e780 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2024-04-27T04:28:52.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.599+0000 7f2f8410e780 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2024-04-27T04:28:52.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.599+0000 7f2f8410e780 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2024-04-27T04:28:52.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.600+0000 7f2f7c8b0700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 94346968208896 notifier 4250 bl.length()=194 2024-04-27T04:28:52.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.600+0000 7f2f7c8b0700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2024-04-27T04:28:52.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.600+0000 7f2f7c8b0700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2024-04-27T04:28:52.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.601+0000 7f2f8410e780 10 cache put: name=default.rgw.meta+users.keys+YQJFAXEUESMVBGWOCZSU info.flags=0x7 2024-04-27T04:28:52.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.601+0000 7f2f8410e780 10 moving default.rgw.meta+users.keys+YQJFAXEUESMVBGWOCZSU to cache LRU end 2024-04-27T04:28:52.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.601+0000 7f2f8410e780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:YQJFAXEUESMVBGWOCZSU, ofs0, ns] 2024-04-27T04:28:52.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.601+0000 7f2f7c0af700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182401 cookie 94346968207744 notifier 4250 bl.length()=186 2024-04-27T04:28:52.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.601+0000 7f2f7c0af700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+YQJFAXEUESMVBGWOCZSU info.flags=0x7 2024-04-27T04:28:52.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.601+0000 7f2f7c0af700 10 rgw watcher librados: moving default.rgw.meta+users.keys+YQJFAXEUESMVBGWOCZSU to cache LRU end 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "user_id": "foo.client.0", 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "display_name": "Mr. foo.client.0", 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "email": "foo.client.0+test@test.test", 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "suspended": 0, 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "max_buckets": 1000, 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "subusers": [], 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "keys": [ 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "user": "foo.client.0", 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "YQJFAXEUESMVBGWOCZSU", 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "GRT8TQ4wLR9qJD3ReMBDYsuQc6GUzz0TfRzuH910LM09ZD3loGLmRw==" 2024-04-27T04:28:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "swift_keys": [], 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": [ 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "user-policy", 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "perm": "*" 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "op_mask": "read, write, delete", 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "", 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "default_storage_class": "", 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_tags": [], 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_quota": { 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2024-04-27T04:28:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "user_quota": { 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "temp_url_keys": [], 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "rgw", 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "mfa_ids": [] 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-04-27T04:28:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:52.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.602+0000 7f2f69089700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:52.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.604+0000 7f2f8410e780 20 remove_watcher() i=0 2024-04-27T04:28:52.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.604+0000 7f2f8410e780 2 removed watcher, disabling cache 2024-04-27T04:28:52.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.605+0000 7f2f8410e780 20 remove_watcher() i=1 2024-04-27T04:28:52.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.606+0000 7f2f8410e780 20 remove_watcher() i=2 2024-04-27T04:28:52.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.607+0000 7f2f8410e780 20 remove_watcher() i=3 2024-04-27T04:28:52.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.608+0000 7f2f8410e780 20 remove_watcher() i=4 2024-04-27T04:28:52.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.609+0000 7f2f8410e780 20 remove_watcher() i=5 2024-04-27T04:28:52.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.610+0000 7f2f8410e780 20 remove_watcher() i=6 2024-04-27T04:28:52.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.611+0000 7f2f8410e780 20 remove_watcher() i=7 2024-04-27T04:28:52.615 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 2343150779 --totp-seed 4YOVK3M7CQX3W63GXGGZDHZVYLKWUCVH342CWQRL2E3U64XQMJXS4NU3RYUSR6C5 --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-04-27T04:28:52.667 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:52.667 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:52.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.675+0000 7f00ae59d780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.676+0000 7f00ae59d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.676+0000 7f00ae59d780 20 realm 2024-04-27T04:28:52.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.676+0000 7f00ae59d780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.677+0000 7f00ae59d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.677+0000 7f00ae59d780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:28:52.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.677+0000 7f00ae59d780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.677+0000 7f00ae59d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.677+0000 7f00ae59d780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.677+0000 7f00ae59d780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:52.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.677+0000 7f00ae59d780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.678+0000 7f00ae59d780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:28:52.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.678+0000 7f00ae59d780 20 searching for the correct realm 2024-04-27T04:28:52.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:28:52.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:28:52.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:28:52.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:28:52.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:52.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 zone default found 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 4 Realm: () 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 zonegroup default 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.685+0000 7f00ae59d780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.686+0000 7f00ae59d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.686+0000 7f00ae59d780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.686+0000 7f00ae59d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.686+0000 7f00ae59d780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.686+0000 7f00ae59d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.686+0000 7f00ae59d780 20 started sync module instance, tier type = 2024-04-27T04:28:52.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.686+0000 7f00ae59d780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:28:52.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.697+0000 7f00ae59d780 20 add_watcher() i=0 2024-04-27T04:28:52.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.697+0000 7f00ae59d780 20 add_watcher() i=1 2024-04-27T04:28:52.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.697+0000 7f00ae59d780 20 add_watcher() i=2 2024-04-27T04:28:52.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.697+0000 7f00ae59d780 20 add_watcher() i=3 2024-04-27T04:28:52.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.697+0000 7f00ae59d780 20 add_watcher() i=4 2024-04-27T04:28:52.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.697+0000 7f00ae59d780 20 add_watcher() i=5 2024-04-27T04:28:52.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.697+0000 7f00ae59d780 20 add_watcher() i=6 2024-04-27T04:28:52.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.698+0000 7f00ae59d780 20 add_watcher() i=7 2024-04-27T04:28:52.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.698+0000 7f00ae59d780 2 all 8 watchers are set, enabling cache 2024-04-27T04:28:52.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.699+0000 7f0093518700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:52.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.699+0000 7f00ae59d780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:52.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.699+0000 7f00ae59d780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:52.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.703+0000 7f00ae59d780 5 note: GC not initialized 2024-04-27T04:28:52.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.704+0000 7f008ed0f700 20 reqs_thread_entry: start 2024-04-27T04:28:52.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.704+0000 7f00ae59d780 20 init_complete bucket index max shards: 11 2024-04-27T04:28:52.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.704+0000 7f00ae59d780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:28:52.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.704+0000 7f008cd0b700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:29:22 2024 2024-04-27T04:28:52.705 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:28:52.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.704+0000 7f008c4e9700 20 reqs_thread_entry: start 2024-04-27T04:28:52.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.704+0000 7f00ae59d780 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-04-27T04:28:52.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.704+0000 7f00ae59d780 20 rados->read ofs=0 len=0 2024-04-27T04:28:52.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.705+0000 7f00ae59d780 20 rados_obj.operate() r=0 bl.length=455 2024-04-27T04:28:52.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.705+0000 7f00ae59d780 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2024-04-27T04:28:52.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.705+0000 7f00ae59d780 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-04-27T04:28:52.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:52.705+0000 7f00ae59d780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T04:28:54.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.685+0000 7f00ae59d780 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-04-27T04:28:54.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.685+0000 7f00ae59d780 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-04-27T04:28:54.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.685+0000 7f00ae59d780 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-04-27T04:28:54.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.685+0000 7f00a653e700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 94535782308608 notifier 4258 bl.length()=654 2024-04-27T04:28:54.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.685+0000 7f00a653e700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-04-27T04:28:54.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.685+0000 7f00a653e700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-04-27T04:28:54.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.687+0000 7f0093518700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:54.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.692+0000 7f00ae59d780 20 remove_watcher() i=0 2024-04-27T04:28:54.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.692+0000 7f00ae59d780 2 removed watcher, disabling cache 2024-04-27T04:28:54.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.693+0000 7f00ae59d780 20 remove_watcher() i=1 2024-04-27T04:28:54.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.695+0000 7f00ae59d780 20 remove_watcher() i=2 2024-04-27T04:28:54.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.696+0000 7f00ae59d780 20 remove_watcher() i=3 2024-04-27T04:28:54.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.698+0000 7f00ae59d780 20 remove_watcher() i=4 2024-04-27T04:28:54.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.702+0000 7f00ae59d780 20 remove_watcher() i=5 2024-04-27T04:28:54.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.703+0000 7f00ae59d780 20 remove_watcher() i=6 2024-04-27T04:28:54.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.708+0000 7f00ae59d780 20 remove_watcher() i=7 2024-04-27T04:28:54.713 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2024-04-27T04:28:54.713 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --email bar.client.0+test@test.test --caps 'user-policy=*' --access-key AIUAGBATEWSPLPONMUBB --secret UbdFzBYOXmji4dVhb2VvgsZI0HtcWCRvnNZM7URkRFOiyGXzme1mCw== --cluster ceph 2024-04-27T04:28:54.768 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:54.768 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:54.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.778+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.779+0000 7ff60a0ee780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.779+0000 7ff60a0ee780 20 realm 2024-04-27T04:28:54.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.779+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.779+0000 7ff60a0ee780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.779+0000 7ff60a0ee780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:28:54.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.779+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.779+0000 7ff60a0ee780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.779+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.779+0000 7ff60a0ee780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:54.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.779+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.780+0000 7ff60a0ee780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:28:54.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.780+0000 7ff60a0ee780 20 searching for the correct realm 2024-04-27T04:28:54.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:28:54.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:28:54.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:28:54.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:28:54.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:54.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:28:54.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 zone default found 2024-04-27T04:28:54.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 4 Realm: () 2024-04-27T04:28:54.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:28:54.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:28:54.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:28:54.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 zonegroup default 2024-04-27T04:28:54.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.788+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.789+0000 7ff60a0ee780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.789+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.789+0000 7ff60a0ee780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.789+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.789+0000 7ff60a0ee780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.789+0000 7ff60a0ee780 20 started sync module instance, tier type = 2024-04-27T04:28:54.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.789+0000 7ff60a0ee780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:28:54.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.815+0000 7ff60a0ee780 20 add_watcher() i=0 2024-04-27T04:28:54.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.815+0000 7ff60a0ee780 20 add_watcher() i=1 2024-04-27T04:28:54.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.815+0000 7ff60a0ee780 20 add_watcher() i=2 2024-04-27T04:28:54.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.815+0000 7ff60a0ee780 20 add_watcher() i=3 2024-04-27T04:28:54.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.815+0000 7ff60a0ee780 20 add_watcher() i=4 2024-04-27T04:28:54.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.815+0000 7ff60a0ee780 20 add_watcher() i=5 2024-04-27T04:28:54.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.815+0000 7ff60a0ee780 20 add_watcher() i=6 2024-04-27T04:28:54.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.818+0000 7ff60a0ee780 20 add_watcher() i=7 2024-04-27T04:28:54.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.818+0000 7ff60a0ee780 2 all 8 watchers are set, enabling cache 2024-04-27T04:28:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.821+0000 7ff5ef069700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.821+0000 7ff60a0ee780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.821+0000 7ff60a0ee780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:54.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.824+0000 7ff60a0ee780 5 note: GC not initialized 2024-04-27T04:28:54.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.825+0000 7ff5ea860700 20 reqs_thread_entry: start 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.825+0000 7ff60a0ee780 20 init_complete bucket index max shards: 11 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.825+0000 7ff60a0ee780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff5e885c700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:29:24 2024 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff5e803a700 20 reqs_thread_entry: start 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2024-04-27T04:28:54.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2024-04-27T04:28:54.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 10 cache get: name=default.rgw.meta+users.keys+AIUAGBATEWSPLPONMUBB : miss 2024-04-27T04:28:54.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 10 cache put: name=default.rgw.meta+users.keys+AIUAGBATEWSPLPONMUBB info.flags=0x0 2024-04-27T04:28:54.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 10 adding default.rgw.meta+users.keys+AIUAGBATEWSPLPONMUBB to cache LRU end 2024-04-27T04:28:54.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 10 cache get: name=default.rgw.meta+users.keys+AIUAGBATEWSPLPONMUBB : hit (negative entry) 2024-04-27T04:28:54.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.826+0000 7ff60a0ee780 10 cache get: name=default.rgw.meta+users.keys+AIUAGBATEWSPLPONMUBB : hit (negative entry) 2024-04-27T04:28:54.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.827+0000 7ff60a0ee780 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-04-27T04:28:54.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.827+0000 7ff60a0ee780 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-04-27T04:28:54.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.827+0000 7ff60a0ee780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-04-27T04:28:54.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.828+0000 7ff602890700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116993 cookie 93911952752512 notifier 4266 bl.length()=640 2024-04-27T04:28:54.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.828+0000 7ff602890700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-04-27T04:28:54.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.828+0000 7ff602890700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-04-27T04:28:54.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.829+0000 7ff60a0ee780 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2024-04-27T04:28:54.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.829+0000 7ff60a0ee780 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2024-04-27T04:28:54.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.829+0000 7ff60a0ee780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2024-04-27T04:28:54.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.829+0000 7ff60208f700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 93911952752512 notifier 4266 bl.length()=194 2024-04-27T04:28:54.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.829+0000 7ff60208f700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2024-04-27T04:28:54.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.829+0000 7ff60208f700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2024-04-27T04:28:54.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.832+0000 7ff60a0ee780 10 cache put: name=default.rgw.meta+users.keys+AIUAGBATEWSPLPONMUBB info.flags=0x7 2024-04-27T04:28:54.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.832+0000 7ff60a0ee780 10 moving default.rgw.meta+users.keys+AIUAGBATEWSPLPONMUBB to cache LRU end 2024-04-27T04:28:54.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.832+0000 7ff60a0ee780 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.keys:AIUAGBATEWSPLPONMUBB, ofs0, ns] 2024-04-27T04:28:54.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.832+0000 7ff602890700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 93911952758272 notifier 4266 bl.length()=186 2024-04-27T04:28:54.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.832+0000 7ff602890700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+AIUAGBATEWSPLPONMUBB info.flags=0x7 2024-04-27T04:28:54.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.832+0000 7ff602890700 10 rgw watcher librados: moving default.rgw.meta+users.keys+AIUAGBATEWSPLPONMUBB to cache LRU end 2024-04-27T04:28:54.832 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-04-27T04:28:54.832 INFO:teuthology.orchestra.run.smithi161.stdout: "user_id": "bar.client.0", 2024-04-27T04:28:54.832 INFO:teuthology.orchestra.run.smithi161.stdout: "display_name": "Mr. bar.client.0", 2024-04-27T04:28:54.832 INFO:teuthology.orchestra.run.smithi161.stdout: "email": "bar.client.0+test@test.test", 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: "suspended": 0, 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: "max_buckets": 1000, 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: "subusers": [], 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: "keys": [ 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: "user": "bar.client.0", 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "AIUAGBATEWSPLPONMUBB", 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "UbdFzBYOXmji4dVhb2VvgsZI0HtcWCRvnNZM7URkRFOiyGXzme1mCw==" 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: "swift_keys": [], 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": [ 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "user-policy", 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: "perm": "*" 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:54.833 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "op_mask": "read, write, delete", 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "", 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "default_storage_class": "", 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_tags": [], 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_quota": { 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "user_quota": { 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2024-04-27T04:28:54.834 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2024-04-27T04:28:54.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:54.835 INFO:teuthology.orchestra.run.smithi161.stdout: "temp_url_keys": [], 2024-04-27T04:28:54.835 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "rgw", 2024-04-27T04:28:54.835 INFO:teuthology.orchestra.run.smithi161.stdout: "mfa_ids": [] 2024-04-27T04:28:54.835 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-04-27T04:28:54.835 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:54.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.833+0000 7ff5ef069700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:54.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.836+0000 7ff60a0ee780 20 remove_watcher() i=0 2024-04-27T04:28:54.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.836+0000 7ff60a0ee780 2 removed watcher, disabling cache 2024-04-27T04:28:54.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.839+0000 7ff60a0ee780 20 remove_watcher() i=1 2024-04-27T04:28:54.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.841+0000 7ff60a0ee780 20 remove_watcher() i=2 2024-04-27T04:28:54.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.842+0000 7ff60a0ee780 20 remove_watcher() i=3 2024-04-27T04:28:54.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.845+0000 7ff60a0ee780 20 remove_watcher() i=4 2024-04-27T04:28:54.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.846+0000 7ff60a0ee780 20 remove_watcher() i=5 2024-04-27T04:28:54.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.847+0000 7ff60a0ee780 20 remove_watcher() i=6 2024-04-27T04:28:54.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.848+0000 7ff60a0ee780 20 remove_watcher() i=7 2024-04-27T04:28:54.852 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 1560519528 --totp-seed FLOMUCIW7T7KPW2YRJH5IDGGEZT5MX72TBJZ6RMWQQSSWREYIEVYWSRK4IXG5VFU --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-04-27T04:28:54.903 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:54.903 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:54.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.912+0000 7f86493fa780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.912+0000 7f86493fa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.912+0000 7f86493fa780 20 realm 2024-04-27T04:28:54.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.912+0000 7f86493fa780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.913+0000 7f86493fa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.913+0000 7f86493fa780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:28:54.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.913+0000 7f86493fa780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.913+0000 7f86493fa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.913+0000 7f86493fa780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.913+0000 7f86493fa780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:54.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.913+0000 7f86493fa780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.913+0000 7f86493fa780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:28:54.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.913+0000 7f86493fa780 20 searching for the correct realm 2024-04-27T04:28:54.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.920+0000 7f86493fa780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.920+0000 7f86493fa780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.920+0000 7f86493fa780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.920+0000 7f86493fa780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.920+0000 7f86493fa780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.920+0000 7f86493fa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.920+0000 7f86493fa780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 20 zone default found 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 4 Realm: () 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:28:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:28:54.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:28:54.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 20 zonegroup default 2024-04-27T04:28:54.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.921+0000 7f86493fa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:54.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.922+0000 7f86493fa780 20 started sync module instance, tier type = 2024-04-27T04:28:54.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.922+0000 7f86493fa780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:28:54.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.935+0000 7f86493fa780 20 add_watcher() i=0 2024-04-27T04:28:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.935+0000 7f86493fa780 20 add_watcher() i=1 2024-04-27T04:28:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.935+0000 7f86493fa780 20 add_watcher() i=2 2024-04-27T04:28:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.935+0000 7f86493fa780 20 add_watcher() i=3 2024-04-27T04:28:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.935+0000 7f86493fa780 20 add_watcher() i=4 2024-04-27T04:28:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.935+0000 7f86493fa780 20 add_watcher() i=5 2024-04-27T04:28:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.935+0000 7f86493fa780 20 add_watcher() i=6 2024-04-27T04:28:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.936+0000 7f86493fa780 20 add_watcher() i=7 2024-04-27T04:28:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.936+0000 7f86493fa780 2 all 8 watchers are set, enabling cache 2024-04-27T04:28:54.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.938+0000 7f862e375700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:54.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.938+0000 7f86493fa780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:54.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.938+0000 7f86493fa780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.941+0000 7f86493fa780 5 note: GC not initialized 2024-04-27T04:28:54.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.942+0000 7f8629b6c700 20 reqs_thread_entry: start 2024-04-27T04:28:54.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.942+0000 7f86493fa780 20 init_complete bucket index max shards: 11 2024-04-27T04:28:54.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.942+0000 7f86493fa780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:28:54.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.942+0000 7f8627b68700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:29:25 2024 2024-04-27T04:28:54.943 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:28:54.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.942+0000 7f8627346700 20 reqs_thread_entry: start 2024-04-27T04:28:54.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.942+0000 7f86493fa780 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-04-27T04:28:54.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.942+0000 7f86493fa780 20 rados->read ofs=0 len=0 2024-04-27T04:28:54.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.943+0000 7f86493fa780 20 rados_obj.operate() r=0 bl.length=455 2024-04-27T04:28:54.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.943+0000 7f86493fa780 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2024-04-27T04:28:54.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.943+0000 7f86493fa780 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-04-27T04:28:54.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.943+0000 7f86493fa780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T04:28:54.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.945+0000 7f86493fa780 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-04-27T04:28:54.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.945+0000 7f86493fa780 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-04-27T04:28:54.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.945+0000 7f86493fa780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-04-27T04:28:54.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.945+0000 7f8641b9c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94148158521216 notifier 4274 bl.length()=654 2024-04-27T04:28:54.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.945+0000 7f8641b9c700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-04-27T04:28:54.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.945+0000 7f8641b9c700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-04-27T04:28:54.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.946+0000 7f862e375700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:54.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.953+0000 7f86493fa780 20 remove_watcher() i=0 2024-04-27T04:28:54.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.953+0000 7f86493fa780 2 removed watcher, disabling cache 2024-04-27T04:28:54.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.954+0000 7f86493fa780 20 remove_watcher() i=1 2024-04-27T04:28:54.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.955+0000 7f86493fa780 20 remove_watcher() i=2 2024-04-27T04:28:54.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.958+0000 7f86493fa780 20 remove_watcher() i=3 2024-04-27T04:28:54.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.960+0000 7f86493fa780 20 remove_watcher() i=4 2024-04-27T04:28:54.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.961+0000 7f86493fa780 20 remove_watcher() i=5 2024-04-27T04:28:54.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.964+0000 7f86493fa780 20 remove_watcher() i=6 2024-04-27T04:28:54.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:54.965+0000 7f86493fa780 20 remove_watcher() i=7 2024-04-27T04:28:54.969 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2024-04-27T04:28:54.969 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --access-key IRKXDLVTKJYWTXALVOZM --secret yzD8xd2QxGa6YNbLJbKWFu9syx5gtwsTMZgHyT52HBAvbgObycpmrA== --cluster ceph 2024-04-27T04:28:55.021 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:55.021 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:55.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.029+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.030+0000 7f06bd998780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.030+0000 7f06bd998780 20 realm 2024-04-27T04:28:55.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.030+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.030+0000 7f06bd998780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.030+0000 7f06bd998780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:28:55.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.030+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.030+0000 7f06bd998780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.030+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.031+0000 7f06bd998780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:55.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.031+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.031+0000 7f06bd998780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:28:55.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.031+0000 7f06bd998780 20 searching for the correct realm 2024-04-27T04:28:55.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.038+0000 7f06bd998780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:28:55.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.038+0000 7f06bd998780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.038+0000 7f06bd998780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.038+0000 7f06bd998780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.038+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.038+0000 7f06bd998780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.038+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 zone default found 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 4 Realm: () 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 zonegroup default 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 started sync module instance, tier type = 2024-04-27T04:28:55.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.039+0000 7f06bd998780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:28:55.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.072+0000 7f06bd998780 20 add_watcher() i=0 2024-04-27T04:28:55.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.072+0000 7f06bd998780 20 add_watcher() i=1 2024-04-27T04:28:55.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.072+0000 7f06bd998780 20 add_watcher() i=2 2024-04-27T04:28:55.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.072+0000 7f06bd998780 20 add_watcher() i=3 2024-04-27T04:28:55.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.072+0000 7f06bd998780 20 add_watcher() i=4 2024-04-27T04:28:55.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.072+0000 7f06bd998780 20 add_watcher() i=5 2024-04-27T04:28:55.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.072+0000 7f06bd998780 20 add_watcher() i=6 2024-04-27T04:28:55.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.074+0000 7f06bd998780 20 add_watcher() i=7 2024-04-27T04:28:55.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.074+0000 7f06bd998780 2 all 8 watchers are set, enabling cache 2024-04-27T04:28:55.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.077+0000 7f06a2913700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:55.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.077+0000 7f06bd998780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:55.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.077+0000 7f06bd998780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:55.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.081+0000 7f06bd998780 5 note: GC not initialized 2024-04-27T04:28:55.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f069e10a700 20 reqs_thread_entry: start 2024-04-27T04:28:55.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 20 init_complete bucket index max shards: 11 2024-04-27T04:28:55.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:28:55.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f069c106700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:29:25 2024 2024-04-27T04:28:55.083 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:28:55.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f069b8e4700 20 reqs_thread_entry: start 2024-04-27T04:28:55.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-04-27T04:28:55.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2024-04-27T04:28:55.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-04-27T04:28:55.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2024-04-27T04:28:55.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2024-04-27T04:28:55.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2024-04-27T04:28:55.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 10 cache get: name=default.rgw.meta+users.keys+IRKXDLVTKJYWTXALVOZM : miss 2024-04-27T04:28:55.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.082+0000 7f06bd998780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.083+0000 7f06bd998780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.083+0000 7f06bd998780 10 cache put: name=default.rgw.meta+users.keys+IRKXDLVTKJYWTXALVOZM info.flags=0x0 2024-04-27T04:28:55.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.083+0000 7f06bd998780 10 adding default.rgw.meta+users.keys+IRKXDLVTKJYWTXALVOZM to cache LRU end 2024-04-27T04:28:55.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.083+0000 7f06bd998780 10 cache get: name=default.rgw.meta+users.keys+IRKXDLVTKJYWTXALVOZM : hit (negative entry) 2024-04-27T04:28:55.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.083+0000 7f06bd998780 10 cache get: name=default.rgw.meta+users.keys+IRKXDLVTKJYWTXALVOZM : hit (negative entry) 2024-04-27T04:28:55.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.084+0000 7f06bd998780 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-04-27T04:28:55.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.084+0000 7f06bd998780 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-04-27T04:28:55.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.084+0000 7f06bd998780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-04-27T04:28:55.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.084+0000 7f06b613a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 94264368845184 notifier 4282 bl.length()=714 2024-04-27T04:28:55.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.084+0000 7f06b613a700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-04-27T04:28:55.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.084+0000 7f06b613a700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-04-27T04:28:55.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.086+0000 7f06bd998780 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2024-04-27T04:28:55.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.086+0000 7f06bd998780 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2024-04-27T04:28:55.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.086+0000 7f06bd998780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2024-04-27T04:28:55.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.087+0000 7f06b5939700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94264368847488 notifier 4282 bl.length()=224 2024-04-27T04:28:55.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.087+0000 7f06b5939700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2024-04-27T04:28:55.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.087+0000 7f06b5939700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2024-04-27T04:28:55.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.089+0000 7f06bd998780 10 cache put: name=default.rgw.meta+users.keys+IRKXDLVTKJYWTXALVOZM info.flags=0x7 2024-04-27T04:28:55.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.089+0000 7f06bd998780 10 moving default.rgw.meta+users.keys+IRKXDLVTKJYWTXALVOZM to cache LRU end 2024-04-27T04:28:55.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.089+0000 7f06bd998780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:IRKXDLVTKJYWTXALVOZM, ofs0, ns] 2024-04-27T04:28:55.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.089+0000 7f06b613a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116997 cookie 94264368840576 notifier 4282 bl.length()=201 2024-04-27T04:28:55.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.089+0000 7f06b613a700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+IRKXDLVTKJYWTXALVOZM info.flags=0x7 2024-04-27T04:28:55.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.089+0000 7f06b613a700 10 rgw watcher librados: moving default.rgw.meta+users.keys+IRKXDLVTKJYWTXALVOZM to cache LRU end 2024-04-27T04:28:55.090 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-04-27T04:28:55.090 INFO:teuthology.orchestra.run.smithi161.stdout: "user_id": "testx$tenanteduser.client.0", 2024-04-27T04:28:55.090 INFO:teuthology.orchestra.run.smithi161.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2024-04-27T04:28:55.090 INFO:teuthology.orchestra.run.smithi161.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2024-04-27T04:28:55.090 INFO:teuthology.orchestra.run.smithi161.stdout: "suspended": 0, 2024-04-27T04:28:55.090 INFO:teuthology.orchestra.run.smithi161.stdout: "max_buckets": 1000, 2024-04-27T04:28:55.090 INFO:teuthology.orchestra.run.smithi161.stdout: "subusers": [], 2024-04-27T04:28:55.090 INFO:teuthology.orchestra.run.smithi161.stdout: "keys": [ 2024-04-27T04:28:55.090 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:55.090 INFO:teuthology.orchestra.run.smithi161.stdout: "user": "testx$tenanteduser.client.0", 2024-04-27T04:28:55.090 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "IRKXDLVTKJYWTXALVOZM", 2024-04-27T04:28:55.090 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "yzD8xd2QxGa6YNbLJbKWFu9syx5gtwsTMZgHyT52HBAvbgObycpmrA==" 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: "swift_keys": [], 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": [ 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "user-policy", 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: "perm": "*" 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: "op_mask": "read, write, delete", 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "", 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: "default_storage_class": "", 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_tags": [], 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_quota": { 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2024-04-27T04:28:55.091 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: "user_quota": { 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: "temp_url_keys": [], 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "rgw", 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: "mfa_ids": [] 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-04-27T04:28:55.092 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:55.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.090+0000 7f06a2913700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:55.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.094+0000 7f06bd998780 20 remove_watcher() i=0 2024-04-27T04:28:55.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.094+0000 7f06bd998780 2 removed watcher, disabling cache 2024-04-27T04:28:55.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.097+0000 7f06bd998780 20 remove_watcher() i=1 2024-04-27T04:28:55.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.099+0000 7f06bd998780 20 remove_watcher() i=2 2024-04-27T04:28:55.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.102+0000 7f06bd998780 20 remove_watcher() i=3 2024-04-27T04:28:55.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.105+0000 7f06bd998780 20 remove_watcher() i=4 2024-04-27T04:28:55.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.107+0000 7f06bd998780 20 remove_watcher() i=5 2024-04-27T04:28:55.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.110+0000 7f06bd998780 20 remove_watcher() i=6 2024-04-27T04:28:55.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.111+0000 7f06bd998780 20 remove_watcher() i=7 2024-04-27T04:28:55.115 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 2985775412 --totp-seed 2WY2LOY53Z3WZY6ZEDJXDBMJNKAH2DB2YMQBPTMHONJOB45OMNT2SUHLWG2GDL3O --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-04-27T04:28:55.167 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:55.167 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:55.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.175+0000 7f07f0e6a780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.176+0000 7f07f0e6a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.176+0000 7f07f0e6a780 20 realm 2024-04-27T04:28:55.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.176+0000 7f07f0e6a780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.176+0000 7f07f0e6a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.176+0000 7f07f0e6a780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:28:55.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.176+0000 7f07f0e6a780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.176+0000 7f07f0e6a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.176+0000 7f07f0e6a780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.177+0000 7f07f0e6a780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:55.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.177+0000 7f07f0e6a780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.177+0000 7f07f0e6a780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:28:55.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.177+0000 7f07f0e6a780 20 searching for the correct realm 2024-04-27T04:28:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:28:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:28:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:28:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:28:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 zone default found 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 4 Realm: () 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 zonegroup default 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.184+0000 7f07f0e6a780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.185+0000 7f07f0e6a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.185+0000 7f07f0e6a780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.185+0000 7f07f0e6a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.185+0000 7f07f0e6a780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.185+0000 7f07f0e6a780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.185+0000 7f07f0e6a780 20 started sync module instance, tier type = 2024-04-27T04:28:55.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.185+0000 7f07f0e6a780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:28:55.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.200+0000 7f07f0e6a780 20 add_watcher() i=0 2024-04-27T04:28:55.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.200+0000 7f07f0e6a780 20 add_watcher() i=1 2024-04-27T04:28:55.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.200+0000 7f07f0e6a780 20 add_watcher() i=2 2024-04-27T04:28:55.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.200+0000 7f07f0e6a780 20 add_watcher() i=3 2024-04-27T04:28:55.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.200+0000 7f07f0e6a780 20 add_watcher() i=4 2024-04-27T04:28:55.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.200+0000 7f07f0e6a780 20 add_watcher() i=5 2024-04-27T04:28:55.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.200+0000 7f07f0e6a780 20 add_watcher() i=6 2024-04-27T04:28:55.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.201+0000 7f07f0e6a780 20 add_watcher() i=7 2024-04-27T04:28:55.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.201+0000 7f07f0e6a780 2 all 8 watchers are set, enabling cache 2024-04-27T04:28:55.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.202+0000 7f07d5de5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:55.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.202+0000 7f07f0e6a780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:55.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.202+0000 7f07f0e6a780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:55.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.206+0000 7f07f0e6a780 5 note: GC not initialized 2024-04-27T04:28:55.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.207+0000 7f07d15dc700 20 reqs_thread_entry: start 2024-04-27T04:28:55.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.207+0000 7f07f0e6a780 20 init_complete bucket index max shards: 11 2024-04-27T04:28:55.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.207+0000 7f07f0e6a780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:28:55.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.207+0000 7f07cf5d8700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:29:25 2024 2024-04-27T04:28:55.207 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:28:55.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.207+0000 7f07cedb6700 20 reqs_thread_entry: start 2024-04-27T04:28:55.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.207+0000 7f07f0e6a780 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-04-27T04:28:55.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.207+0000 7f07f0e6a780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.208+0000 7f07f0e6a780 20 rados_obj.operate() r=0 bl.length=514 2024-04-27T04:28:55.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.208+0000 7f07f0e6a780 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2024-04-27T04:28:55.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.208+0000 7f07f0e6a780 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-04-27T04:28:55.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.208+0000 7f07f0e6a780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T04:28:55.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.210+0000 7f07f0e6a780 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-04-27T04:28:55.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.210+0000 7f07f0e6a780 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-04-27T04:28:55.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.210+0000 7f07f0e6a780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-04-27T04:28:55.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.210+0000 7f07e960c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116997 cookie 94748739744128 notifier 4290 bl.length()=728 2024-04-27T04:28:55.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.210+0000 7f07e960c700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-04-27T04:28:55.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.210+0000 7f07e960c700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-04-27T04:28:55.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.211+0000 7f07d5de5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:55.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.213+0000 7f07f0e6a780 20 remove_watcher() i=0 2024-04-27T04:28:55.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.213+0000 7f07f0e6a780 2 removed watcher, disabling cache 2024-04-27T04:28:55.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.214+0000 7f07f0e6a780 20 remove_watcher() i=1 2024-04-27T04:28:55.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.215+0000 7f07f0e6a780 20 remove_watcher() i=2 2024-04-27T04:28:55.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.216+0000 7f07f0e6a780 20 remove_watcher() i=3 2024-04-27T04:28:55.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.217+0000 7f07f0e6a780 20 remove_watcher() i=4 2024-04-27T04:28:55.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.217+0000 7f07f0e6a780 20 remove_watcher() i=5 2024-04-27T04:28:55.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.218+0000 7f07f0e6a780 20 remove_watcher() i=6 2024-04-27T04:28:55.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.219+0000 7f07f0e6a780 20 remove_watcher() i=7 2024-04-27T04:28:55.223 DEBUG:tasks.s3tests:Creating user foobar.client.0 on client.0 2024-04-27T04:28:55.223 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foobar.client.0 --display-name 'Mr. foobar.client.0' --email foobar.client.0+test@test.test --caps 'user-policy=*' --access-key XAWGBWOAXEXRCJJYOYMT --secret EKvjrduiHVXj5uIiL226kIJq1azI9qi0ib5INMHaBbXYpa4sv8YShQ== --cluster ceph 2024-04-27T04:28:55.275 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:55.275 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:55.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.284+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.285+0000 7f99ff684780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.285+0000 7f99ff684780 20 realm 2024-04-27T04:28:55.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.285+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.285+0000 7f99ff684780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.285+0000 7f99ff684780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:28:55.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.285+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.285+0000 7f99ff684780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.285+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.285+0000 7f99ff684780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:55.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.285+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.286+0000 7f99ff684780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:28:55.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.286+0000 7f99ff684780 20 searching for the correct realm 2024-04-27T04:28:55.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:28:55.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:28:55.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:28:55.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:28:55.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:55.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 zone default found 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 4 Realm: () 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 zonegroup default 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.293+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.294+0000 7f99ff684780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.294+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.294+0000 7f99ff684780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.294+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.294+0000 7f99ff684780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.294+0000 7f99ff684780 20 started sync module instance, tier type = 2024-04-27T04:28:55.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.294+0000 7f99ff684780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:28:55.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.310+0000 7f99ff684780 20 add_watcher() i=0 2024-04-27T04:28:55.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.310+0000 7f99ff684780 20 add_watcher() i=1 2024-04-27T04:28:55.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.310+0000 7f99ff684780 20 add_watcher() i=2 2024-04-27T04:28:55.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.310+0000 7f99ff684780 20 add_watcher() i=3 2024-04-27T04:28:55.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.310+0000 7f99ff684780 20 add_watcher() i=4 2024-04-27T04:28:55.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.310+0000 7f99ff684780 20 add_watcher() i=5 2024-04-27T04:28:55.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.310+0000 7f99ff684780 20 add_watcher() i=6 2024-04-27T04:28:55.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.311+0000 7f99ff684780 20 add_watcher() i=7 2024-04-27T04:28:55.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.311+0000 7f99ff684780 2 all 8 watchers are set, enabling cache 2024-04-27T04:28:55.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.312+0000 7f99e45ff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:55.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.312+0000 7f99ff684780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:55.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.312+0000 7f99ff684780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:55.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.315+0000 7f99ff684780 5 note: GC not initialized 2024-04-27T04:28:55.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.316+0000 7f99dfdf6700 20 reqs_thread_entry: start 2024-04-27T04:28:55.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.316+0000 7f99ff684780 20 init_complete bucket index max shards: 11 2024-04-27T04:28:55.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.316+0000 7f99ff684780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.316+0000 7f99dddf2700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:29:25 2024 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.316+0000 7f99dd5d0700 20 reqs_thread_entry: start 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x0 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 10 cache get: name=default.rgw.meta+users.email+foobar.client.0+test@test.test : miss 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 10 cache put: name=default.rgw.meta+users.email+foobar.client.0+test@test.test info.flags=0x0 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 10 adding default.rgw.meta+users.email+foobar.client.0+test@test.test to cache LRU end 2024-04-27T04:28:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 10 cache get: name=default.rgw.meta+users.keys+XAWGBWOAXEXRCJJYOYMT : miss 2024-04-27T04:28:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 10 cache put: name=default.rgw.meta+users.keys+XAWGBWOAXEXRCJJYOYMT info.flags=0x0 2024-04-27T04:28:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 10 adding default.rgw.meta+users.keys+XAWGBWOAXEXRCJJYOYMT to cache LRU end 2024-04-27T04:28:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 10 cache get: name=default.rgw.meta+users.keys+XAWGBWOAXEXRCJJYOYMT : hit (negative entry) 2024-04-27T04:28:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.317+0000 7f99ff684780 10 cache get: name=default.rgw.meta+users.keys+XAWGBWOAXEXRCJJYOYMT : hit (negative entry) 2024-04-27T04:28:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.318+0000 7f99ff684780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-04-27T04:28:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.318+0000 7f99ff684780 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:28:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.318+0000 7f99ff684780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2024-04-27T04:28:55.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.319+0000 7f99f7e26700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116998 cookie 94403565034112 notifier 4298 bl.length()=655 2024-04-27T04:28:55.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.319+0000 7f99f7e26700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-04-27T04:28:55.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.319+0000 7f99f7e26700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:28:55.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.323+0000 7f99ff684780 10 cache put: name=default.rgw.meta+users.email+foobar.client.0+test@test.test info.flags=0x7 2024-04-27T04:28:55.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.323+0000 7f99ff684780 10 moving default.rgw.meta+users.email+foobar.client.0+test@test.test to cache LRU end 2024-04-27T04:28:55.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.323+0000 7f99ff684780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.email:foobar.client.0+test@test.test, ofs0, ns] 2024-04-27T04:28:55.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.323+0000 7f99f7e26700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116999 cookie 94403565031808 notifier 4298 bl.length()=200 2024-04-27T04:28:55.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.323+0000 7f99f7e26700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foobar.client.0+test@test.test info.flags=0x7 2024-04-27T04:28:55.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.323+0000 7f99f7e26700 10 rgw watcher librados: moving default.rgw.meta+users.email+foobar.client.0+test@test.test to cache LRU end 2024-04-27T04:28:55.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.325+0000 7f99ff684780 10 cache put: name=default.rgw.meta+users.keys+XAWGBWOAXEXRCJJYOYMT info.flags=0x7 2024-04-27T04:28:55.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.325+0000 7f99ff684780 10 moving default.rgw.meta+users.keys+XAWGBWOAXEXRCJJYOYMT to cache LRU end 2024-04-27T04:28:55.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.325+0000 7f99ff684780 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.keys:XAWGBWOAXEXRCJJYOYMT, ofs0, ns] 2024-04-27T04:28:55.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.325+0000 7f99f7e26700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117000 cookie 94403565026048 notifier 4298 bl.length()=189 2024-04-27T04:28:55.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.325+0000 7f99f7e26700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+XAWGBWOAXEXRCJJYOYMT info.flags=0x7 2024-04-27T04:28:55.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.325+0000 7f99f7e26700 10 rgw watcher librados: moving default.rgw.meta+users.keys+XAWGBWOAXEXRCJJYOYMT to cache LRU end 2024-04-27T04:28:55.325 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-04-27T04:28:55.325 INFO:teuthology.orchestra.run.smithi161.stdout: "user_id": "foobar.client.0", 2024-04-27T04:28:55.325 INFO:teuthology.orchestra.run.smithi161.stdout: "display_name": "Mr. foobar.client.0", 2024-04-27T04:28:55.325 INFO:teuthology.orchestra.run.smithi161.stdout: "email": "foobar.client.0+test@test.test", 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: "suspended": 0, 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: "max_buckets": 1000, 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: "subusers": [], 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: "keys": [ 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: "user": "foobar.client.0", 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "XAWGBWOAXEXRCJJYOYMT", 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "EKvjrduiHVXj5uIiL226kIJq1azI9qi0ib5INMHaBbXYpa4sv8YShQ==" 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: "swift_keys": [], 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": [ 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "user-policy", 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: "perm": "*" 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:55.326 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "op_mask": "read, write, delete", 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "", 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "default_storage_class": "", 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_tags": [], 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_quota": { 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "user_quota": { 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2024-04-27T04:28:55.327 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2024-04-27T04:28:55.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:55.328 INFO:teuthology.orchestra.run.smithi161.stdout: "temp_url_keys": [], 2024-04-27T04:28:55.328 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "rgw", 2024-04-27T04:28:55.328 INFO:teuthology.orchestra.run.smithi161.stdout: "mfa_ids": [] 2024-04-27T04:28:55.328 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-04-27T04:28:55.328 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:55.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.326+0000 7f99e45ff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:55.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.330+0000 7f99ff684780 20 remove_watcher() i=0 2024-04-27T04:28:55.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.330+0000 7f99ff684780 2 removed watcher, disabling cache 2024-04-27T04:28:55.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.332+0000 7f99ff684780 20 remove_watcher() i=1 2024-04-27T04:28:55.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.333+0000 7f99ff684780 20 remove_watcher() i=2 2024-04-27T04:28:55.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.336+0000 7f99ff684780 20 remove_watcher() i=3 2024-04-27T04:28:55.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.339+0000 7f99ff684780 20 remove_watcher() i=4 2024-04-27T04:28:55.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.340+0000 7f99ff684780 20 remove_watcher() i=5 2024-04-27T04:28:55.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.343+0000 7f99ff684780 20 remove_watcher() i=6 2024-04-27T04:28:55.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.346+0000 7f99ff684780 20 remove_watcher() i=7 2024-04-27T04:28:55.350 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foobar.client.0 --totp-serial 6344563966 --totp-seed QCNEMIDCZZTXQ773K2R65SAXMQILVHMTW3F7SKESD234PN56FN53J65Z7XWIUMXP --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-04-27T04:28:55.401 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:55.401 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:55.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.409+0000 7ff9078d5780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.410+0000 7ff9078d5780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.410+0000 7ff9078d5780 20 realm 2024-04-27T04:28:55.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.410+0000 7ff9078d5780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.410+0000 7ff9078d5780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.410+0000 7ff9078d5780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:28:55.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.410+0000 7ff9078d5780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.410+0000 7ff9078d5780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.410+0000 7ff9078d5780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.410+0000 7ff9078d5780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:55.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.410+0000 7ff9078d5780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.411+0000 7ff9078d5780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:28:55.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.411+0000 7ff9078d5780 20 searching for the correct realm 2024-04-27T04:28:55.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:28:55.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:28:55.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:28:55.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:28:55.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:55.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 zone default found 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 4 Realm: () 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 zonegroup default 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.418+0000 7ff9078d5780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.419+0000 7ff9078d5780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.419+0000 7ff9078d5780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.419+0000 7ff9078d5780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.419+0000 7ff9078d5780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.419+0000 7ff9078d5780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.419+0000 7ff9078d5780 20 started sync module instance, tier type = 2024-04-27T04:28:55.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.419+0000 7ff9078d5780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:28:55.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.447+0000 7ff9078d5780 20 add_watcher() i=0 2024-04-27T04:28:55.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.447+0000 7ff9078d5780 20 add_watcher() i=1 2024-04-27T04:28:55.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.447+0000 7ff9078d5780 20 add_watcher() i=2 2024-04-27T04:28:55.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.447+0000 7ff9078d5780 20 add_watcher() i=3 2024-04-27T04:28:55.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.447+0000 7ff9078d5780 20 add_watcher() i=4 2024-04-27T04:28:55.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.447+0000 7ff9078d5780 20 add_watcher() i=5 2024-04-27T04:28:55.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.447+0000 7ff9078d5780 20 add_watcher() i=6 2024-04-27T04:28:55.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.450+0000 7ff9078d5780 20 add_watcher() i=7 2024-04-27T04:28:55.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.450+0000 7ff9078d5780 2 all 8 watchers are set, enabling cache 2024-04-27T04:28:55.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.453+0000 7ff8ec850700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:55.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.453+0000 7ff9078d5780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:55.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.453+0000 7ff9078d5780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:55.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.456+0000 7ff9078d5780 5 note: GC not initialized 2024-04-27T04:28:55.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.457+0000 7ff8e8047700 20 reqs_thread_entry: start 2024-04-27T04:28:55.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.457+0000 7ff9078d5780 20 init_complete bucket index max shards: 11 2024-04-27T04:28:55.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.457+0000 7ff9078d5780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:28:55.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.457+0000 7ff8e6043700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:29:25 2024 2024-04-27T04:28:55.458 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:28:55.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.457+0000 7ff8e5821700 20 reqs_thread_entry: start 2024-04-27T04:28:55.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.457+0000 7ff9078d5780 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2024-04-27T04:28:55.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.457+0000 7ff9078d5780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.458+0000 7ff9078d5780 20 rados_obj.operate() r=0 bl.length=467 2024-04-27T04:28:55.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.458+0000 7ff9078d5780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2024-04-27T04:28:55.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.458+0000 7ff9078d5780 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:28:55.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.458+0000 7ff9078d5780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T04:28:55.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.460+0000 7ff9078d5780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-04-27T04:28:55.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.460+0000 7ff9078d5780 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:28:55.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.460+0000 7ff9078d5780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2024-04-27T04:28:55.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.460+0000 7ff900077700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117001 cookie 94423996316288 notifier 4306 bl.length()=669 2024-04-27T04:28:55.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.460+0000 7ff900077700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-04-27T04:28:55.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.460+0000 7ff900077700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:28:55.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.461+0000 7ff8ec850700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:55.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.465+0000 7ff9078d5780 20 remove_watcher() i=0 2024-04-27T04:28:55.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.465+0000 7ff9078d5780 2 removed watcher, disabling cache 2024-04-27T04:28:55.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.468+0000 7ff9078d5780 20 remove_watcher() i=1 2024-04-27T04:28:55.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.471+0000 7ff9078d5780 20 remove_watcher() i=2 2024-04-27T04:28:55.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.473+0000 7ff9078d5780 20 remove_watcher() i=3 2024-04-27T04:28:55.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.474+0000 7ff9078d5780 20 remove_watcher() i=4 2024-04-27T04:28:55.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.475+0000 7ff9078d5780 20 remove_watcher() i=5 2024-04-27T04:28:55.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.479+0000 7ff9078d5780 20 remove_watcher() i=6 2024-04-27T04:28:55.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.480+0000 7ff9078d5780 20 remove_watcher() i=7 2024-04-27T04:28:55.484 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 caps add --uid foobar.client.0 --caps 'roles=*' --cluster ceph 2024-04-27T04:28:55.528 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:55.528 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:55.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.537+0000 7f859abae780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.537+0000 7f859abae780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.537+0000 7f859abae780 20 realm 2024-04-27T04:28:55.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.537+0000 7f859abae780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.538+0000 7f859abae780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.538+0000 7f859abae780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:28:55.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.538+0000 7f859abae780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.538+0000 7f859abae780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.538+0000 7f859abae780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.538+0000 7f859abae780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:55.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.538+0000 7f859abae780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.538+0000 7f859abae780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:28:55.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.538+0000 7f859abae780 20 searching for the correct realm 2024-04-27T04:28:55.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.545+0000 7f859abae780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.545+0000 7f859abae780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.545+0000 7f859abae780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.545+0000 7f859abae780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.545+0000 7f859abae780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.545+0000 7f859abae780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.545+0000 7f859abae780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 zone default found 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 4 Realm: () 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 zonegroup default 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 started sync module instance, tier type = 2024-04-27T04:28:55.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.546+0000 7f859abae780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:28:55.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.564+0000 7f859abae780 20 add_watcher() i=0 2024-04-27T04:28:55.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.564+0000 7f859abae780 20 add_watcher() i=1 2024-04-27T04:28:55.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.564+0000 7f859abae780 20 add_watcher() i=2 2024-04-27T04:28:55.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.564+0000 7f859abae780 20 add_watcher() i=3 2024-04-27T04:28:55.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.564+0000 7f859abae780 20 add_watcher() i=4 2024-04-27T04:28:55.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.564+0000 7f859abae780 20 add_watcher() i=5 2024-04-27T04:28:55.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.564+0000 7f859abae780 20 add_watcher() i=6 2024-04-27T04:28:55.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.565+0000 7f859abae780 20 add_watcher() i=7 2024-04-27T04:28:55.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.565+0000 7f859abae780 2 all 8 watchers are set, enabling cache 2024-04-27T04:28:55.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.566+0000 7f857fb29700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:55.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.566+0000 7f859abae780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:55.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.566+0000 7f859abae780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:55.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.570+0000 7f859abae780 5 note: GC not initialized 2024-04-27T04:28:55.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.571+0000 7f857b320700 20 reqs_thread_entry: start 2024-04-27T04:28:55.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.571+0000 7f859abae780 20 init_complete bucket index max shards: 11 2024-04-27T04:28:55.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.571+0000 7f859abae780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:28:55.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.571+0000 7f857931c700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:29:26 2024 2024-04-27T04:28:55.572 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:28:55.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.571+0000 7f8578afa700 20 reqs_thread_entry: start 2024-04-27T04:28:55.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.571+0000 7f859abae780 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2024-04-27T04:28:55.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.571+0000 7f859abae780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.571+0000 7f859abae780 20 rados_obj.operate() r=0 bl.length=481 2024-04-27T04:28:55.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.572+0000 7f859abae780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2024-04-27T04:28:55.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.572+0000 7f859abae780 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:28:55.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.572+0000 7f859abae780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T04:28:55.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.573+0000 7f859abae780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-04-27T04:28:55.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.573+0000 7f859abae780 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:28:55.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.573+0000 7f859abae780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2024-04-27T04:28:55.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.573+0000 7f8593350700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117002 cookie 94163512064640 notifier 4314 bl.length()=682 2024-04-27T04:28:55.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.573+0000 7f8593350700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-04-27T04:28:55.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.573+0000 7f8593350700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:28:55.573 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-04-27T04:28:55.573 INFO:teuthology.orchestra.run.smithi161.stdout: "user_id": "foobar.client.0", 2024-04-27T04:28:55.573 INFO:teuthology.orchestra.run.smithi161.stdout: "display_name": "Mr. foobar.client.0", 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: "email": "foobar.client.0+test@test.test", 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: "suspended": 0, 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: "max_buckets": 1000, 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: "subusers": [], 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: "keys": [ 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: "user": "foobar.client.0", 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "XAWGBWOAXEXRCJJYOYMT", 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "EKvjrduiHVXj5uIiL226kIJq1azI9qi0ib5INMHaBbXYpa4sv8YShQ==" 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: "swift_keys": [], 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": [ 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "roles", 2024-04-27T04:28:55.574 INFO:teuthology.orchestra.run.smithi161.stdout: "perm": "*" 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "user-policy", 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: "perm": "*" 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: "op_mask": "read, write, delete", 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "", 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: "default_storage_class": "", 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_tags": [], 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_quota": { 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2024-04-27T04:28:55.575 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: "user_quota": { 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: "temp_url_keys": [], 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "rgw", 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: "mfa_ids": [ 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: "6344563966" 2024-04-27T04:28:55.576 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-04-27T04:28:55.577 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-04-27T04:28:55.577 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:55.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.574+0000 7f857fb29700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:55.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.576+0000 7f859abae780 20 remove_watcher() i=0 2024-04-27T04:28:55.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.576+0000 7f859abae780 2 removed watcher, disabling cache 2024-04-27T04:28:55.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.577+0000 7f859abae780 20 remove_watcher() i=1 2024-04-27T04:28:55.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.578+0000 7f859abae780 20 remove_watcher() i=2 2024-04-27T04:28:55.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.579+0000 7f859abae780 20 remove_watcher() i=3 2024-04-27T04:28:55.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.580+0000 7f859abae780 20 remove_watcher() i=4 2024-04-27T04:28:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.580+0000 7f859abae780 20 remove_watcher() i=5 2024-04-27T04:28:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.581+0000 7f859abae780 20 remove_watcher() i=6 2024-04-27T04:28:55.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.582+0000 7f859abae780 20 remove_watcher() i=7 2024-04-27T04:28:55.586 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 caps add --uid foobar.client.0 --caps 'oidc-provider=*' --cluster ceph 2024-04-27T04:28:55.631 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:28:55.631 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:28:55.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.640+0000 7fdd11903780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.641+0000 7fdd11903780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.641+0000 7fdd11903780 20 realm 2024-04-27T04:28:55.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.641+0000 7fdd11903780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.641+0000 7fdd11903780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.641+0000 7fdd11903780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:28:55.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.641+0000 7fdd11903780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.641+0000 7fdd11903780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.641+0000 7fdd11903780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.641+0000 7fdd11903780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:55.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.641+0000 7fdd11903780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.642+0000 7fdd11903780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:28:55.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.642+0000 7fdd11903780 20 searching for the correct realm 2024-04-27T04:28:55.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:28:55.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:28:55.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:28:55.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:28:55.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:28:55.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 zone default found 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 4 Realm: () 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 zonegroup default 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.649+0000 7fdd11903780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.650+0000 7fdd11903780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.650+0000 7fdd11903780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.650+0000 7fdd11903780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.650+0000 7fdd11903780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.650+0000 7fdd11903780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.650+0000 7fdd11903780 20 started sync module instance, tier type = 2024-04-27T04:28:55.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.650+0000 7fdd11903780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:28:55.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.658+0000 7fdd11903780 20 add_watcher() i=0 2024-04-27T04:28:55.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.658+0000 7fdd11903780 20 add_watcher() i=1 2024-04-27T04:28:55.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.658+0000 7fdd11903780 20 add_watcher() i=2 2024-04-27T04:28:55.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.658+0000 7fdd11903780 20 add_watcher() i=3 2024-04-27T04:28:55.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.658+0000 7fdd11903780 20 add_watcher() i=4 2024-04-27T04:28:55.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.658+0000 7fdd11903780 20 add_watcher() i=5 2024-04-27T04:28:55.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.658+0000 7fdd11903780 20 add_watcher() i=6 2024-04-27T04:28:55.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.659+0000 7fdd11903780 20 add_watcher() i=7 2024-04-27T04:28:55.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.659+0000 7fdd11903780 2 all 8 watchers are set, enabling cache 2024-04-27T04:28:55.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.660+0000 7fdcf687e700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:55.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.660+0000 7fdd11903780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:28:55.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.660+0000 7fdd11903780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:28:55.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.667+0000 7fdd11903780 5 note: GC not initialized 2024-04-27T04:28:55.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.668+0000 7fdcf2075700 20 reqs_thread_entry: start 2024-04-27T04:28:55.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.668+0000 7fdd11903780 20 init_complete bucket index max shards: 11 2024-04-27T04:28:55.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.668+0000 7fdd11903780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:28:55.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.668+0000 7fdd11903780 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2024-04-27T04:28:55.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.668+0000 7fdd11903780 20 rados->read ofs=0 len=0 2024-04-27T04:28:55.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.669+0000 7fdcef870700 20 reqs_thread_entry: start 2024-04-27T04:28:55.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.669+0000 7fdd11903780 20 rados_obj.operate() r=0 bl.length=494 2024-04-27T04:28:55.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.669+0000 7fdd11903780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2024-04-27T04:28:55.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.669+0000 7fdd11903780 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:28:55.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.669+0000 7fdd11903780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T04:28:55.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.670+0000 7fdcf0071700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:29:26 2024 2024-04-27T04:28:55.670 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:28:55.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.671+0000 7fdd11903780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-04-27T04:28:55.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.671+0000 7fdd11903780 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:28:55.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.671+0000 7fdd11903780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2024-04-27T04:28:55.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.672+0000 7fdd0a0a5700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117003 cookie 94771451042432 notifier 4322 bl.length()=703 2024-04-27T04:28:55.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.672+0000 7fdd0a0a5700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2024-04-27T04:28:55.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.672+0000 7fdd0a0a5700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:28:55.677 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-04-27T04:28:55.677 INFO:teuthology.orchestra.run.smithi161.stdout: "user_id": "foobar.client.0", 2024-04-27T04:28:55.677 INFO:teuthology.orchestra.run.smithi161.stdout: "display_name": "Mr. foobar.client.0", 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: "email": "foobar.client.0+test@test.test", 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: "suspended": 0, 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: "max_buckets": 1000, 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: "subusers": [], 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: "keys": [ 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: "user": "foobar.client.0", 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "XAWGBWOAXEXRCJJYOYMT", 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "EKvjrduiHVXj5uIiL226kIJq1azI9qi0ib5INMHaBbXYpa4sv8YShQ==" 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: "swift_keys": [], 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": [ 2024-04-27T04:28:55.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "oidc-provider", 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: "perm": "*" 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "roles", 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: "perm": "*" 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "user-policy", 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: "perm": "*" 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: "op_mask": "read, write, delete", 2024-04-27T04:28:55.679 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "", 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "default_storage_class": "", 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_tags": [], 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_quota": { 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "user_quota": { 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-04-27T04:28:55.680 INFO:teuthology.orchestra.run.smithi161.stdout: "temp_url_keys": [], 2024-04-27T04:28:55.681 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "rgw", 2024-04-27T04:28:55.681 INFO:teuthology.orchestra.run.smithi161.stdout: "mfa_ids": [ 2024-04-27T04:28:55.681 INFO:teuthology.orchestra.run.smithi161.stdout: "6344563966" 2024-04-27T04:28:55.681 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-04-27T04:28:55.681 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-04-27T04:28:55.681 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:28:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.679+0000 7fdcf687e700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:28:55.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.683+0000 7fdd11903780 20 remove_watcher() i=0 2024-04-27T04:28:55.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.683+0000 7fdd11903780 2 removed watcher, disabling cache 2024-04-27T04:28:55.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.684+0000 7fdd11903780 20 remove_watcher() i=1 2024-04-27T04:28:55.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.685+0000 7fdd11903780 20 remove_watcher() i=2 2024-04-27T04:28:55.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.686+0000 7fdd11903780 20 remove_watcher() i=3 2024-04-27T04:28:55.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.687+0000 7fdd11903780 20 remove_watcher() i=4 2024-04-27T04:28:55.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.688+0000 7fdd11903780 20 remove_watcher() i=5 2024-04-27T04:28:55.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.688+0000 7fdd11903780 20 remove_watcher() i=6 2024-04-27T04:28:55.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:28:55.689+0000 7fdd11903780 20 remove_watcher() i=7 2024-04-27T04:28:55.693 INFO:tasks.s3tests:Configuring s3-tests... 2024-04-27T04:28:55.694 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:55.694 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2024-04-27T04:28:55.711 INFO:tasks.s3tests:Configuring boto... 2024-04-27T04:28:55.711 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-27T04:28:55.711 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/boto-client.0.cfg 2024-04-27T04:28:55.767 DEBUG:teuthology.orchestra.run.smithi161:s3 tests against rgw> source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m lifecycle 2024-04-27T04:28:55.987 INFO:teuthology.orchestra.run.smithi161.stdout:GLOB sdist-make: /home/ubuntu/cephtest/s3-tests-client.0/setup.py 2024-04-27T04:28:56.184 INFO:teuthology.orchestra.run.smithi161.stdout:py create: /home/ubuntu/cephtest/s3-tests-client.0/.tox/py 2024-04-27T04:28:56.649 INFO:teuthology.orchestra.run.smithi161.stdout:py installdeps: -rrequirements.txt 2024-04-27T04:29:14.303 INFO:teuthology.orchestra.run.smithi161.stdout:py inst: /home/ubuntu/cephtest/s3-tests-client.0/.tox/.tmp/package/1/s3tests-0.0.1.zip 2024-04-27T04:29:16.205 INFO:teuthology.orchestra.run.smithi161.stdout:py installed: attrs==22.2.0,boto==2.49.0,boto3==1.23.10,botocore==1.26.10,certifi==2024.2.2,charset-normalizer==2.0.12,distlib==0.3.8,filelock==3.4.1,gevent==22.10.2,greenlet==2.0.2,httplib2==0.22.0,idna==3.7,importlib-metadata==4.8.3,importlib-resources==5.4.0,iniconfig==1.1.1,isodate==0.6.1,jmespath==0.10.0,lxml==5.2.1,munch==4.0.0,packaging==21.3,platformdirs==2.4.0,pluggy==1.0.0,py==1.11.0,pyparsing==3.1.2,pytest==7.0.1,python-dateutil==2.9.0.post0,pytz==2024.1,PyYAML==6.0.1,requests==2.27.1,s3tests @ file:///home/ubuntu/cephtest/s3-tests-client.0/.tox/.tmp/package/1/s3tests-0.0.1.zip,s3transfer==0.5.2,six==1.16.0,toml==0.10.2,tomli==1.2.3,tox==3.28.0,typing_extensions==4.1.1,urllib3==1.26.18,virtualenv==20.17.1,zipp==3.6.0,zope.event==4.6,zope.interface==5.5.2 2024-04-27T04:29:16.205 INFO:teuthology.orchestra.run.smithi161.stdout:py run-test-pre: PYTHONHASHSEED='3259130163' 2024-04-27T04:29:16.205 INFO:teuthology.orchestra.run.smithi161.stdout:py run-test: commands[0] | pytest -v -m lifecycle 2024-04-27T04:29:16.433 INFO:teuthology.orchestra.run.smithi161.stdout:============================= test session starts ============================== 2024-04-27T04:29:16.433 INFO:teuthology.orchestra.run.smithi161.stdout:platform linux -- Python 3.6.8, pytest-7.0.1, pluggy-1.0.0 -- /home/ubuntu/cephtest/s3-tests-client.0/.tox/py/bin/python 2024-04-27T04:29:16.433 INFO:teuthology.orchestra.run.smithi161.stdout:cachedir: .tox/py/.pytest_cache 2024-04-27T04:29:16.433 INFO:teuthology.orchestra.run.smithi161.stdout:rootdir: /home/ubuntu/cephtest/s3-tests-client.0, configfile: pytest.ini 2024-04-27T04:29:17.967 INFO:teuthology.orchestra.run.smithi161.stdout:collecting ... collected 778 items / 740 deselected / 38 selected 2024-04-27T04:29:17.968 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:29:18.188 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set PASSED [ 2%] 2024-04-27T04:29:18.358 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_get PASSED [ 5%] 2024-04-27T04:29:18.529 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_get_no_id PASSED [ 7%] 2024-04-27T04:30:28.926 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration PASSED [ 10%] 2024-04-27T04:31:39.258 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecyclev2_expiration PASSED [ 13%] 2024-04-27T04:32:09.549 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_versioning_enabled PASSED [ 15%] 2024-04-27T04:32:39.704 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_tags1 PASSED [ 18%] 2024-04-27T04:33:09.941 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_tags2 PASSED [ 21%] 2024-04-27T04:33:40.223 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_versioned_tags2 PASSED [ 23%] 2024-04-27T04:34:50.678 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_noncur_tags1 PASSED [ 26%] 2024-04-27T04:34:50.808 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_id_too_long PASSED [ 28%] 2024-04-27T04:34:50.969 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_same_id PASSED [ 31%] 2024-04-27T04:34:51.182 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_invalid_status PASSED [ 34%] 2024-04-27T04:34:51.311 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_date PASSED [ 36%] 2024-04-27T04:34:51.443 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_invalid_date PASSED [ 39%] 2024-04-27T04:35:21.715 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_date PASSED [ 42%] 2024-04-27T04:35:21.904 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_days0 PASSED [ 44%] 2024-04-27T04:35:22.088 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_header_put PASSED [ 47%] 2024-04-27T04:35:22.271 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_header_head PASSED [ 50%] 2024-04-27T04:35:22.502 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_header_tags_head PASSED [ 52%] 2024-04-27T04:35:22.689 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_header_and_tags_head PASSED [ 55%] 2024-04-27T04:35:22.874 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_noncurrent PASSED [ 57%] 2024-04-27T04:36:13.265 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_noncur_expiration PASSED [ 60%] 2024-04-27T04:36:13.395 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_deletemarker PASSED [ 63%] 2024-04-27T04:36:13.528 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_filter PASSED [ 65%] 2024-04-27T04:36:13.659 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_empty_filter PASSED [ 68%] 2024-04-27T04:37:24.033 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_deletemarker_expiration PASSED [ 71%] 2024-04-27T04:37:24.204 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_multipart PASSED [ 73%] 2024-04-27T04:38:15.673 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_multipart_expiration PASSED [ 76%] 2024-04-27T04:38:15.808 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_transition_set_invalid_date PASSED [ 78%] 2024-04-27T04:39:56.173 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_transition PASSED [ 81%] 2024-04-27T04:41:56.549 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_transition_single_rule_multi_trans PASSED [ 84%] 2024-04-27T04:41:56.681 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_set_noncurrent_transition PASSED [ 86%] 2024-04-27T04:44:17.156 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_noncur_transition PASSED [ 89%] 2024-04-27T04:44:17.189 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_cloud_transition SKIPPED [ 92%] 2024-04-27T04:44:17.220 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_cloud_multiple_transition SKIPPED [ 94%] 2024-04-27T04:44:17.285 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_noncur_cloud_transition SKIPPED [ 97%] 2024-04-27T04:44:17.319 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py::test_lifecycle_cloud_transition_large_obj SKIPPED [100%] 2024-04-27T04:44:17.320 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.320 INFO:teuthology.orchestra.run.smithi161.stdout:=============================== warnings summary =============================== 2024-04-27T04:44:17.320 INFO:teuthology.orchestra.run.smithi161.stdout:.tox/py/lib/python3.6/site-packages/boto/plugin.py:40 2024-04-27T04:44:17.320 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/.tox/py/lib/python3.6/site-packages/boto/plugin.py:40: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses 2024-04-27T04:44:17.320 INFO:teuthology.orchestra.run.smithi161.stdout: import imp 2024-04-27T04:44:17.320 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.320 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests/functional/test_s3.py:359 2024-04-27T04:44:17.320 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests/functional/test_s3.py:359: DeprecationWarning: invalid escape sequence \W 2024-04-27T04:44:17.320 INFO:teuthology.orchestra.run.smithi161.stdout: extra_sc = re.split('\W+', config['main']['storage_classes']) 2024-04-27T04:44:17.320 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.320 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests/functional/test_s3_website.py:253 2024-04-27T04:44:17.320 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests/functional/test_s3_website.py:253: PytestUnknownMarkWarning: Unknown pytest.mark.timeout - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html 2024-04-27T04:44:17.321 INFO:teuthology.orchestra.run.smithi161.stdout: @pytest.mark.timeout(10) 2024-04-27T04:44:17.321 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.321 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests/functional/test_s3_website.py:1077 2024-04-27T04:44:17.321 INFO:teuthology.orchestra.run.smithi161.stdout: s3tests/functional/test_s3_website.py:1077: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_routing_generator will be ignored 2024-04-27T04:44:17.321 INFO:teuthology.orchestra.run.smithi161.stdout: @pytest.mark.s3website_routing_rules 2024-04-27T04:44:17.321 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.321 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py:2166 2024-04-27T04:44:17.321 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2166: DeprecationWarning: invalid escape sequence \$ 2024-04-27T04:44:17.321 INFO:teuthology.orchestra.run.smithi161.stdout: ["starts-with", "$key", "\$foo"],\ 2024-04-27T04:44:17.321 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.321 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py:2181 2024-04-27T04:44:17.321 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2181: DeprecationWarning: invalid escape sequence \$ 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout: payload = OrderedDict([ ("key" , "\$foo.txt"),("AWSAccessKeyId" , aws_access_key_id),\ 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py:2187 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2187: DeprecationWarning: invalid escape sequence \$ 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout: response = client.get_object(Bucket=bucket_name, Key='\$foo.txt') 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py:2244 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2244: DeprecationWarning: invalid escape sequence \$ 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout: ["starts-with", "$key", "\$foo"],\ 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py:2259 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2259: DeprecationWarning: invalid escape sequence \$ 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout: payload = OrderedDict([ ("key" , "\$foo.txt"),("AWSAccessKeyId" , aws_access_key_id),\ 2024-04-27T04:44:17.322 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py:2277 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2277: DeprecationWarning: invalid escape sequence \$ 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout: ["starts-with", "$key", "\$foo"],\ 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py:2292 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2292: DeprecationWarning: invalid escape sequence \$ 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout: payload = OrderedDict([ ("key" , "\$foo.txt"),("AWSAccessKeyId" , aws_access_key_id[::-1]),\ 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py:2310 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2310: DeprecationWarning: invalid escape sequence \$ 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout: ["starts-with", "$key", "\$foo"],\ 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.323 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py:2325 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2325: DeprecationWarning: invalid escape sequence \$ 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout: payload = OrderedDict([ ("key" , "\$foo.txt"),("AWSAccessKeyId" , aws_access_key_id),\ 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py:2375 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2375: DeprecationWarning: invalid escape sequence \$ 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout: ["starts-with", "$key", "\$foo"],\ 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py:2407 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/s3tests_boto3/functional/test_s3.py:2407: DeprecationWarning: invalid escape sequence \$ 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout: ["starts-with", "$key", "\$foo"],\ 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout:s3tests_boto3/functional/test_s3.py: 38 warnings 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout: /home/ubuntu/cephtest/s3-tests-client.0/.tox/py/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2024-04-27T04:44:17.324 INFO:teuthology.orchestra.run.smithi161.stdout: warnings.warn(warning, PythonDeprecationWarning) 2024-04-27T04:44:17.325 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-27T04:44:17.325 INFO:teuthology.orchestra.run.smithi161.stdout:-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2024-04-27T04:44:17.325 INFO:teuthology.orchestra.run.smithi161.stdout:==== 34 passed, 4 skipped, 740 deselected, 53 warnings in 900.88s (0:15:00) ==== 2024-04-27T04:44:17.427 INFO:teuthology.orchestra.run.smithi161.stdout:___________________________________ summary ____________________________________ 2024-04-27T04:44:17.427 INFO:teuthology.orchestra.run.smithi161.stdout: py: commands succeeded 2024-04-27T04:44:17.427 INFO:teuthology.orchestra.run.smithi161.stdout: congratulations :) 2024-04-27T04:44:17.452 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2024-04-27T04:44:17.453 DEBUG:teuthology.orchestra.run.smithi161:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2024-04-27T04:44:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T04:44:17.474 INFO:tasks.s3tests:Cleaning up boto... 2024-04-27T04:44:17.474 DEBUG:teuthology.orchestra.run.smithi161:> rm /home/ubuntu/cephtest/boto-client.0.cfg 2024-04-27T04:44:17.531 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2024-04-27T04:44:17.614 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:44:17.614 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:44:17.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.622+0000 7f9c9eeb2780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.624+0000 7f9c9eeb2780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.624+0000 7f9c9eeb2780 20 realm 2024-04-27T04:44:17.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.624+0000 7f9c9eeb2780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.624+0000 7f9c9eeb2780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.624+0000 7f9c9eeb2780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:44:17.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.624+0000 7f9c9eeb2780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.624+0000 7f9c9eeb2780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.624+0000 7f9c9eeb2780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.624+0000 7f9c9eeb2780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:44:17.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.624+0000 7f9c9eeb2780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.625+0000 7f9c9eeb2780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:44:17.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.625+0000 7f9c9eeb2780 20 searching for the correct realm 2024-04-27T04:44:17.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:44:17.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:44:17.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:44:17.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:44:17.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 zone default found 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 4 Realm: () 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 zonegroup default 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.633+0000 7f9c9eeb2780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.634+0000 7f9c9eeb2780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.634+0000 7f9c9eeb2780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.634+0000 7f9c9eeb2780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.634+0000 7f9c9eeb2780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.634+0000 7f9c9eeb2780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.634+0000 7f9c9eeb2780 20 started sync module instance, tier type = 2024-04-27T04:44:17.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.634+0000 7f9c9eeb2780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:44:17.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.642+0000 7f9c9eeb2780 20 add_watcher() i=0 2024-04-27T04:44:17.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.642+0000 7f9c9eeb2780 20 add_watcher() i=1 2024-04-27T04:44:17.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.642+0000 7f9c9eeb2780 20 add_watcher() i=2 2024-04-27T04:44:17.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.642+0000 7f9c9eeb2780 20 add_watcher() i=3 2024-04-27T04:44:17.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.642+0000 7f9c9eeb2780 20 add_watcher() i=4 2024-04-27T04:44:17.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.642+0000 7f9c9eeb2780 20 add_watcher() i=5 2024-04-27T04:44:17.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.642+0000 7f9c9eeb2780 20 add_watcher() i=6 2024-04-27T04:44:17.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.642+0000 7f9c9eeb2780 20 add_watcher() i=7 2024-04-27T04:44:17.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.642+0000 7f9c9eeb2780 2 all 8 watchers are set, enabling cache 2024-04-27T04:44:17.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.644+0000 7f9c83e2d700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:44:17.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.644+0000 7f9c9eeb2780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:44:17.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.644+0000 7f9c9eeb2780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:44:17.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.689+0000 7f9c7f624700 20 reqs_thread_entry: start 2024-04-27T04:44:17.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.689+0000 7f9c9eeb2780 20 init_complete bucket index max shards: 11 2024-04-27T04:44:17.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.689+0000 7f9c9eeb2780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:44:17.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.689+0000 7f9c7d620700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:44:48 2024 2024-04-27T04:44:17.692 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:44:17.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.689+0000 7f9c7cdfe700 20 reqs_thread_entry: start 2024-04-27T04:44:17.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.689+0000 7f9c9eeb2780 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-04-27T04:44:17.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.689+0000 7f9c9eeb2780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.690+0000 7f9c9eeb2780 20 rados_obj.operate() r=0 bl.length=469 2024-04-27T04:44:17.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.690+0000 7f9c9eeb2780 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2024-04-27T04:44:17.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.690+0000 7f9c9eeb2780 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-04-27T04:44:17.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.690+0000 7f9c9eeb2780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T04:44:17.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.690+0000 7f9c9eeb2780 10 removing key index: YQJFAXEUESMVBGWOCZSU 2024-04-27T04:44:17.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.691+0000 7f9c9eeb2780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.keys:YQJFAXEUESMVBGWOCZSU, ofs0, ns] 2024-04-27T04:44:17.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.691+0000 7f9c97654700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659148 cookie 94739525292416 notifier 4330 bl.length()=170 2024-04-27T04:44:17.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.691+0000 7f9c9eeb2780 10 removing email index: foo.client.0+test@test.test 2024-04-27T04:44:17.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.692+0000 7f9c9eeb2780 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2024-04-27T04:44:17.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.692+0000 7f9c97654700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659297 cookie 94739525293568 notifier 4330 bl.length()=178 2024-04-27T04:44:17.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.693+0000 7f9c9eeb2780 10 removing user buckets index 2024-04-27T04:44:17.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.694+0000 7f9c9eeb2780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2024-04-27T04:44:17.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.694+0000 7f9c97654700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659298 cookie 94739525287808 notifier 4330 bl.length()=169 2024-04-27T04:44:17.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.694+0000 7f9c9eeb2780 10 removing user index: foo.client.0 2024-04-27T04:44:17.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.695+0000 7f9c9eeb2780 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2024-04-27T04:44:17.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.695+0000 7f9c9eeb2780 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-04-27T04:44:17.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.695+0000 7f9c96e53700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659149 cookie 94739525286656 notifier 4330 bl.length()=161 2024-04-27T04:44:17.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.696+0000 7f9c83e2d700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:44:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.698+0000 7f9c9eeb2780 20 remove_watcher() i=0 2024-04-27T04:44:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.698+0000 7f9c9eeb2780 2 removed watcher, disabling cache 2024-04-27T04:44:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.699+0000 7f9c9eeb2780 20 remove_watcher() i=1 2024-04-27T04:44:17.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.700+0000 7f9c9eeb2780 20 remove_watcher() i=2 2024-04-27T04:44:17.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.700+0000 7f9c9eeb2780 20 remove_watcher() i=3 2024-04-27T04:44:17.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.701+0000 7f9c9eeb2780 20 remove_watcher() i=4 2024-04-27T04:44:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.702+0000 7f9c9eeb2780 20 remove_watcher() i=5 2024-04-27T04:44:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.703+0000 7f9c9eeb2780 20 remove_watcher() i=6 2024-04-27T04:44:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.703+0000 7f9c9eeb2780 20 remove_watcher() i=7 2024-04-27T04:44:17.709 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2024-04-27T04:44:17.793 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:44:17.793 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:44:17.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.799+0000 7f861ddb0780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.800+0000 7f861ddb0780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.800+0000 7f861ddb0780 20 realm 2024-04-27T04:44:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.800+0000 7f861ddb0780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.800+0000 7f861ddb0780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.800+0000 7f861ddb0780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:44:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.800+0000 7f861ddb0780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.800+0000 7f861ddb0780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.800+0000 7f861ddb0780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.801+0000 7f861ddb0780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:44:17.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.801+0000 7f861ddb0780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.801+0000 7f861ddb0780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:44:17.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.801+0000 7f861ddb0780 20 searching for the correct realm 2024-04-27T04:44:17.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.808+0000 7f861ddb0780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:44:17.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.808+0000 7f861ddb0780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:44:17.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.808+0000 7f861ddb0780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.808+0000 7f861ddb0780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.808+0000 7f861ddb0780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.808+0000 7f861ddb0780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.808+0000 7f861ddb0780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.808+0000 7f861ddb0780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.808+0000 7f861ddb0780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 20 zone default found 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 4 Realm: () 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 20 zonegroup default 2024-04-27T04:44:17.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 20 started sync module instance, tier type = 2024-04-27T04:44:17.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.809+0000 7f861ddb0780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:44:17.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.816+0000 7f861ddb0780 20 add_watcher() i=0 2024-04-27T04:44:17.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.816+0000 7f861ddb0780 20 add_watcher() i=1 2024-04-27T04:44:17.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.816+0000 7f861ddb0780 20 add_watcher() i=2 2024-04-27T04:44:17.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.816+0000 7f861ddb0780 20 add_watcher() i=3 2024-04-27T04:44:17.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.816+0000 7f861ddb0780 20 add_watcher() i=4 2024-04-27T04:44:17.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.816+0000 7f861ddb0780 20 add_watcher() i=5 2024-04-27T04:44:17.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.816+0000 7f861ddb0780 20 add_watcher() i=6 2024-04-27T04:44:17.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.817+0000 7f861ddb0780 20 add_watcher() i=7 2024-04-27T04:44:17.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.817+0000 7f861ddb0780 2 all 8 watchers are set, enabling cache 2024-04-27T04:44:17.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.819+0000 7f8602d2b700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:44:17.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.819+0000 7f861ddb0780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:44:17.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.819+0000 7f861ddb0780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:44:17.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.863+0000 7f85fe522700 20 reqs_thread_entry: start 2024-04-27T04:44:17.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.864+0000 7f861ddb0780 20 init_complete bucket index max shards: 11 2024-04-27T04:44:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.864+0000 7f861ddb0780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:44:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.864+0000 7f85fc51e700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:44:48 2024 2024-04-27T04:44:17.867 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:44:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.864+0000 7f85fbcfc700 20 reqs_thread_entry: start 2024-04-27T04:44:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.864+0000 7f861ddb0780 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-04-27T04:44:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.864+0000 7f861ddb0780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.864+0000 7f861ddb0780 20 rados_obj.operate() r=0 bl.length=469 2024-04-27T04:44:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.864+0000 7f861ddb0780 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2024-04-27T04:44:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.864+0000 7f861ddb0780 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-04-27T04:44:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.864+0000 7f861ddb0780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T04:44:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.864+0000 7f861ddb0780 10 removing key index: AIUAGBATEWSPLPONMUBB 2024-04-27T04:44:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.865+0000 7f861ddb0780 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.keys:AIUAGBATEWSPLPONMUBB, ofs0, ns] 2024-04-27T04:44:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.865+0000 7f8616552700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659299 cookie 94436820366848 notifier 4338 bl.length()=170 2024-04-27T04:44:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.866+0000 7f861ddb0780 10 removing email index: bar.client.0+test@test.test 2024-04-27T04:44:17.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.867+0000 7f861ddb0780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2024-04-27T04:44:17.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.867+0000 7f8615d51700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659300 cookie 94436820361088 notifier 4338 bl.length()=178 2024-04-27T04:44:17.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.867+0000 7f861ddb0780 10 removing user buckets index 2024-04-27T04:44:17.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.868+0000 7f861ddb0780 10 removing user index: bar.client.0 2024-04-27T04:44:17.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.869+0000 7f861ddb0780 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2024-04-27T04:44:17.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.869+0000 7f861ddb0780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-04-27T04:44:17.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.869+0000 7f8616552700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659301 cookie 94436820361088 notifier 4338 bl.length()=161 2024-04-27T04:44:17.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.870+0000 7f8602d2b700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:44:17.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.872+0000 7f861ddb0780 20 remove_watcher() i=0 2024-04-27T04:44:17.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.872+0000 7f861ddb0780 2 removed watcher, disabling cache 2024-04-27T04:44:17.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.873+0000 7f861ddb0780 20 remove_watcher() i=1 2024-04-27T04:44:17.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.874+0000 7f861ddb0780 20 remove_watcher() i=2 2024-04-27T04:44:17.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.875+0000 7f861ddb0780 20 remove_watcher() i=3 2024-04-27T04:44:17.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.875+0000 7f861ddb0780 20 remove_watcher() i=4 2024-04-27T04:44:17.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.876+0000 7f861ddb0780 20 remove_watcher() i=5 2024-04-27T04:44:17.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.877+0000 7f861ddb0780 20 remove_watcher() i=6 2024-04-27T04:44:17.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.878+0000 7f861ddb0780 20 remove_watcher() i=7 2024-04-27T04:44:17.883 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2024-04-27T04:44:17.967 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:44:17.967 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:44:17.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.973+0000 7f6ff7a3b780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.974+0000 7f6ff7a3b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.974+0000 7f6ff7a3b780 20 realm 2024-04-27T04:44:17.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.974+0000 7f6ff7a3b780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.974+0000 7f6ff7a3b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.974+0000 7f6ff7a3b780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:44:17.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.974+0000 7f6ff7a3b780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.974+0000 7f6ff7a3b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.974+0000 7f6ff7a3b780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.975+0000 7f6ff7a3b780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:44:17.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.975+0000 7f6ff7a3b780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.975+0000 7f6ff7a3b780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:44:17.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.975+0000 7f6ff7a3b780 20 searching for the correct realm 2024-04-27T04:44:17.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:44:17.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:44:17.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:44:17.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:44:17.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:44:17.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:44:17.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 zone default found 2024-04-27T04:44:17.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 4 Realm: () 2024-04-27T04:44:17.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:44:17.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:44:17.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:44:17.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 zonegroup default 2024-04-27T04:44:17.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.982+0000 7f6ff7a3b780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.983+0000 7f6ff7a3b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.983+0000 7f6ff7a3b780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.983+0000 7f6ff7a3b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.983+0000 7f6ff7a3b780 20 rados->read ofs=0 len=0 2024-04-27T04:44:17.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.983+0000 7f6ff7a3b780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:17.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.983+0000 7f6ff7a3b780 20 started sync module instance, tier type = 2024-04-27T04:44:17.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.983+0000 7f6ff7a3b780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:44:17.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.990+0000 7f6ff7a3b780 20 add_watcher() i=0 2024-04-27T04:44:17.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.990+0000 7f6ff7a3b780 20 add_watcher() i=1 2024-04-27T04:44:17.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.990+0000 7f6ff7a3b780 20 add_watcher() i=2 2024-04-27T04:44:17.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.990+0000 7f6ff7a3b780 20 add_watcher() i=3 2024-04-27T04:44:17.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.990+0000 7f6ff7a3b780 20 add_watcher() i=4 2024-04-27T04:44:17.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.990+0000 7f6ff7a3b780 20 add_watcher() i=5 2024-04-27T04:44:17.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.990+0000 7f6ff7a3b780 20 add_watcher() i=6 2024-04-27T04:44:17.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.991+0000 7f6ff7a3b780 20 add_watcher() i=7 2024-04-27T04:44:17.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.991+0000 7f6ff7a3b780 2 all 8 watchers are set, enabling cache 2024-04-27T04:44:17.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.992+0000 7f6fdc9b6700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:44:17.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.992+0000 7f6ff7a3b780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:44:17.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:17.992+0000 7f6ff7a3b780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:44:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.037+0000 7f6fd81ad700 20 reqs_thread_entry: start 2024-04-27T04:44:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.037+0000 7f6ff7a3b780 20 init_complete bucket index max shards: 11 2024-04-27T04:44:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.037+0000 7f6ff7a3b780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:44:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.037+0000 7f6fd61a9700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:44:48 2024 2024-04-27T04:44:18.040 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:44:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.037+0000 7f6fd5987700 20 reqs_thread_entry: start 2024-04-27T04:44:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.037+0000 7f6ff7a3b780 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-04-27T04:44:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.037+0000 7f6ff7a3b780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.038+0000 7f6ff7a3b780 20 rados_obj.operate() r=0 bl.length=528 2024-04-27T04:44:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.038+0000 7f6ff7a3b780 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2024-04-27T04:44:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.038+0000 7f6ff7a3b780 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-04-27T04:44:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.038+0000 7f6ff7a3b780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T04:44:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.038+0000 7f6ff7a3b780 10 removing key index: IRKXDLVTKJYWTXALVOZM 2024-04-27T04:44:18.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.039+0000 7f6ff7a3b780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:IRKXDLVTKJYWTXALVOZM, ofs0, ns] 2024-04-27T04:44:18.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.039+0000 7f6ff01dd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659302 cookie 94105408962432 notifier 4346 bl.length()=170 2024-04-27T04:44:18.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.039+0000 7f6ff7a3b780 10 removing email index: testx$tenanteduser.client.0+test@test.test 2024-04-27T04:44:18.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.040+0000 7f6ff7a3b780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2024-04-27T04:44:18.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.040+0000 7f6fef9dc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659150 cookie 94105408969344 notifier 4346 bl.length()=193 2024-04-27T04:44:18.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.041+0000 7f6ff7a3b780 10 removing user buckets index 2024-04-27T04:44:18.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.041+0000 7f6ff7a3b780 10 removing user index: testx$tenanteduser.client.0 2024-04-27T04:44:18.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.042+0000 7f6ff7a3b780 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2024-04-27T04:44:18.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.042+0000 7f6ff7a3b780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-04-27T04:44:18.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.042+0000 7f6ff01dd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659151 cookie 94105408967040 notifier 4346 bl.length()=176 2024-04-27T04:44:18.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.043+0000 7f6fdc9b6700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:44:18.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.045+0000 7f6ff7a3b780 20 remove_watcher() i=0 2024-04-27T04:44:18.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.045+0000 7f6ff7a3b780 2 removed watcher, disabling cache 2024-04-27T04:44:18.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.046+0000 7f6ff7a3b780 20 remove_watcher() i=1 2024-04-27T04:44:18.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.047+0000 7f6ff7a3b780 20 remove_watcher() i=2 2024-04-27T04:44:18.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.048+0000 7f6ff7a3b780 20 remove_watcher() i=3 2024-04-27T04:44:18.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.048+0000 7f6ff7a3b780 20 remove_watcher() i=4 2024-04-27T04:44:18.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.049+0000 7f6ff7a3b780 20 remove_watcher() i=5 2024-04-27T04:44:18.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.050+0000 7f6ff7a3b780 20 remove_watcher() i=6 2024-04-27T04:44:18.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.051+0000 7f6ff7a3b780 20 remove_watcher() i=7 2024-04-27T04:44:18.057 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foobar.client.0 --purge-data --cluster ceph 2024-04-27T04:44:18.140 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2024-04-27T04:44:18.140 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T04:44:18.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.146+0000 7fa40b83c780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.147+0000 7fa40b83c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:18.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.147+0000 7fa40b83c780 20 realm 2024-04-27T04:44:18.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.147+0000 7fa40b83c780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.147+0000 7fa40b83c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:18.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.147+0000 7fa40b83c780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T04:44:18.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.147+0000 7fa40b83c780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.148+0000 7fa40b83c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:18.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.148+0000 7fa40b83c780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.148+0000 7fa40b83c780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:44:18.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.148+0000 7fa40b83c780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.148+0000 7fa40b83c780 20 rados_obj.operate() r=0 bl.length=1004 2024-04-27T04:44:18.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.148+0000 7fa40b83c780 20 searching for the correct realm 2024-04-27T04:44:18.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.155+0000 7fa40b83c780 20 RGWRados::pool_iterate: got zonegroup_info.ccf87e47-dcb1-4f9a-9e81-afaf69a6803c 2024-04-27T04:44:18.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.155+0000 7fa40b83c780 20 RGWRados::pool_iterate: got zone_info.93e3c4d1-2922-476e-bc51-fb4366c74703 2024-04-27T04:44:18.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.155+0000 7fa40b83c780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T04:44:18.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.155+0000 7fa40b83c780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T04:44:18.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.155+0000 7fa40b83c780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:18.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T04:44:18.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 20 rados_obj.operate() r=0 bl.length=454 2024-04-27T04:44:18.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 20 zone default found 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 4 Realm: () 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 4 ZoneGroup: default (ccf87e47-dcb1-4f9a-9e81-afaf69a6803c) 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 4 Zone: default (93e3c4d1-2922-476e-bc51-fb4366c74703) 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 20 zonegroup default 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.156+0000 7fa40b83c780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.157+0000 7fa40b83c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.157+0000 7fa40b83c780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.157+0000 7fa40b83c780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.157+0000 7fa40b83c780 20 started sync module instance, tier type = 2024-04-27T04:44:18.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.157+0000 7fa40b83c780 20 started zone id=93e3c4d1-2922-476e-bc51-fb4366c74703 (name=default) with tier type = 2024-04-27T04:44:18.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.164+0000 7fa40b83c780 20 add_watcher() i=0 2024-04-27T04:44:18.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.164+0000 7fa40b83c780 20 add_watcher() i=1 2024-04-27T04:44:18.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.164+0000 7fa40b83c780 20 add_watcher() i=2 2024-04-27T04:44:18.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.164+0000 7fa40b83c780 20 add_watcher() i=3 2024-04-27T04:44:18.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.164+0000 7fa40b83c780 20 add_watcher() i=4 2024-04-27T04:44:18.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.164+0000 7fa40b83c780 20 add_watcher() i=5 2024-04-27T04:44:18.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.164+0000 7fa40b83c780 20 add_watcher() i=6 2024-04-27T04:44:18.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.164+0000 7fa40b83c780 20 add_watcher() i=7 2024-04-27T04:44:18.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.164+0000 7fa40b83c780 2 all 8 watchers are set, enabling cache 2024-04-27T04:44:18.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.166+0000 7fa3f07b7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:44:18.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.166+0000 7fa40b83c780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T04:44:18.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.166+0000 7fa40b83c780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-27T04:44:18.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.210+0000 7fa3ebfae700 20 reqs_thread_entry: start 2024-04-27T04:44:18.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.210+0000 7fa40b83c780 20 init_complete bucket index max shards: 11 2024-04-27T04:44:18.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.210+0000 7fa40b83c780 10 rgw notify: Started notification manager with: 1 workers 2024-04-27T04:44:18.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.210+0000 7fa3e9faa700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 27 04:44:48 2024 2024-04-27T04:44:18.213 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-27T04:44:18.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.210+0000 7fa3e9788700 20 reqs_thread_entry: start 2024-04-27T04:44:18.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.210+0000 7fa40b83c780 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2024-04-27T04:44:18.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.210+0000 7fa40b83c780 20 rados->read ofs=0 len=0 2024-04-27T04:44:18.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.210+0000 7fa40b83c780 20 rados_obj.operate() r=0 bl.length=515 2024-04-27T04:44:18.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.210+0000 7fa40b83c780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2024-04-27T04:44:18.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.210+0000 7fa40b83c780 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2024-04-27T04:44:18.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.210+0000 7fa40b83c780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T04:44:18.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.211+0000 7fa40b83c780 10 removing key index: XAWGBWOAXEXRCJJYOYMT 2024-04-27T04:44:18.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.211+0000 7fa40b83c780 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.keys:XAWGBWOAXEXRCJJYOYMT, ofs0, ns] 2024-04-27T04:44:18.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.212+0000 7fa403fde700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659152 cookie 94559551855360 notifier 4354 bl.length()=170 2024-04-27T04:44:18.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.212+0000 7fa40b83c780 10 removing email index: foobar.client.0+test@test.test 2024-04-27T04:44:18.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.213+0000 7fa40b83c780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.email:foobar.client.0+test@test.test, ofs0, ns] 2024-04-27T04:44:18.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.213+0000 7fa403fde700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659153 cookie 94559551861120 notifier 4354 bl.length()=181 2024-04-27T04:44:18.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.213+0000 7fa40b83c780 10 removing user buckets index 2024-04-27T04:44:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.214+0000 7fa40b83c780 10 removing user index: foobar.client.0 2024-04-27T04:44:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.215+0000 7fa40b83c780 10 removing default.rgw.meta+users.uid+foobar.client.0 from cache 2024-04-27T04:44:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.215+0000 7fa40b83c780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2024-04-27T04:44:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.215+0000 7fa4037dd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659154 cookie 94559551863424 notifier 4354 bl.length()=164 2024-04-27T04:44:18.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.216+0000 7fa3f07b7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T04:44:18.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.218+0000 7fa40b83c780 20 remove_watcher() i=0 2024-04-27T04:44:18.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.218+0000 7fa40b83c780 2 removed watcher, disabling cache 2024-04-27T04:44:18.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.219+0000 7fa40b83c780 20 remove_watcher() i=1 2024-04-27T04:44:18.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.220+0000 7fa40b83c780 20 remove_watcher() i=2 2024-04-27T04:44:18.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.220+0000 7fa40b83c780 20 remove_watcher() i=3 2024-04-27T04:44:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.221+0000 7fa40b83c780 20 remove_watcher() i=4 2024-04-27T04:44:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.222+0000 7fa40b83c780 20 remove_watcher() i=5 2024-04-27T04:44:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.223+0000 7fa40b83c780 20 remove_watcher() i=6 2024-04-27T04:44:18.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-27T04:44:18.223+0000 7fa40b83c780 20 remove_watcher() i=7 2024-04-27T04:44:18.229 INFO:tasks.s3tests:Removing s3-tests... 2024-04-27T04:44:18.230 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf /home/ubuntu/cephtest/s3-tests-client.0 2024-04-27T04:44:18.404 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2024-04-27T04:44:18.413 DEBUG:teuthology.run_tasks:Unwinding manager tox 2024-04-27T04:44:18.433 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf /home/ubuntu/cephtest/tox-venv 2024-04-27T04:44:18.475 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-04-27T04:44:18.485 DEBUG:tasks.rgw.client.0:waiting for process to exit 2024-04-27T04:44:18.485 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T04:44:18.516 INFO:tasks.rgw.client.0.smithi161.stdout:2024-04-27T04:44:18.514+0000 7f228f1fd700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground (PID: 45988) UID: 0 2024-04-27T04:44:18.516 INFO:tasks.rgw.client.0.smithi161.stdout:2024-04-27T04:44:18.514+0000 7f22943d27c0 -1 shutting down 2024-04-27T04:44:20.339 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2024-04-27T04:44:24.587 INFO:tasks.rgw.client.0:Stopped 2024-04-27T04:44:24.587 DEBUG:teuthology.orchestra.run.smithi161:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2024-04-27T04:44:24.608 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f /etc/ceph/vault-root-token 2024-04-27T04:44:24.695 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-27T04:44:24.705 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-27T04:44:25.052 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-27T04:44:25.053 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-27T04:44:25.053 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-27T04:44:25.053 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T04:44:25.099 INFO:tasks.ceph.osd.0:Stopped 2024-04-27T04:44:25.099 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-27T04:44:25.099 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T04:44:25.142 INFO:tasks.ceph.osd.1:Stopped 2024-04-27T04:44:25.142 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-27T04:44:25.143 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T04:44:25.194 INFO:tasks.ceph.osd.2:Stopped 2024-04-27T04:44:25.194 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-27T04:44:25.194 DEBUG:tasks.ceph.mgr.0:waiting for process to exit 2024-04-27T04:44:25.194 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T04:44:25.218 INFO:tasks.ceph.mgr.0:Stopped 2024-04-27T04:44:25.218 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-27T04:44:25.218 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-27T04:44:25.219 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T04:44:25.269 INFO:tasks.ceph.mon.a:Stopped 2024-04-27T04:44:25.270 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-27T04:44:25.270 DEBUG:teuthology.orchestra.run.smithi161:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-27T04:44:25.340 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi161.front.sepia.ceph.com 2024-04-27T04:44:25.340 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-27T04:44:25.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2024-04-27T04:44:25.704 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi161.front.sepia.ceph.com 2024-04-27T04:44:25.704 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-27T04:44:25.799 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi161.front.sepia.ceph.com 2024-04-27T04:44:25.799 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-27T04:44:25.864 INFO:tasks.ceph:Archiving mon data... 2024-04-27T04:44:25.865 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-26_22:40:03-rgw-reef-distro-default-smithi/7675694/data/mon.a.tgz 2024-04-27T04:44:25.865 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-04-27T04:44:25.882 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.bWhF4PoAfE 2024-04-27T04:44:25.882 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.bWhF4PoAfE 2024-04-27T04:44:26.028 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.bWhF4PoAfE 2024-04-27T04:44:26.115 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.bWhF4PoAfE is 301KB 2024-04-27T04:44:26.210 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.bWhF4PoAfE 2024-04-27T04:44:26.227 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-27T04:44:26.227 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-27T04:44:26.293 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-27T04:44:26.293 DEBUG:teuthology.orchestra.run.smithi161:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-27T04:44:26.360 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-27T04:44:26.366 INFO:tasks.ceph:Archiving crash dumps... 2024-04-27T04:44:26.368 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-26_22:40:03-rgw-reef-distro-default-smithi/7675694/remote/smithi161/crash 2024-04-27T04:44:26.368 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-27T04:44:26.432 INFO:tasks.ceph:Compressing logs... 2024-04-27T04:44:26.432 DEBUG:teuthology.orchestra.run.smithi161:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T04:44:41.467 INFO:tasks.ceph:Archiving logs... 2024-04-27T04:44:41.467 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-26_22:40:03-rgw-reef-distro-default-smithi/7675694/remote/smithi161/log 2024-04-27T04:44:41.468 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T04:44:43.320 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-27T04:44:43.330 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-27T04:44:43.330 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T04:44:43.384 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-27T04:44:43.394 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-27T04:44:43.394 DEBUG:teuthology.orchestra.run.smithi161:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T04:44:43.415 INFO:teuthology.orchestra.run.smithi161.stderr:bash: ntpq: command not found 2024-04-27T04:44:43.419 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T04:44:43.419 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-04-27T04:44:43.419 INFO:teuthology.orchestra.run.smithi161.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 +636us[ +633us] +/- 59ms 2024-04-27T04:44:43.419 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 10 -1862us[-1865us] +/- 60ms 2024-04-27T04:44:43.419 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +1473us[+1470us] +/- 69ms 2024-04-27T04:44:43.419 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T04:44:43.420 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-27T04:44:43.429 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-27T04:44:43.430 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-27T04:44:43.439 DEBUG:teuthology.orchestra.run.smithi161:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T04:44:43.522 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-27T04:44:43.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T04:44:43.551 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-27T04:44:43.561 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-27T04:44:43.571 INFO:teuthology.task.internal:Duration was 1845.188833 seconds 2024-04-27T04:44:43.571 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-27T04:44:43.611 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-27T04:44:43.611 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T04:44:43.659 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T04:44:43.877 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-27T04:44:43.877 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2024-04-27T04:44:43.877 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T04:44:43.899 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-27T04:44:43.899 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T04:44:43.983 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-27T04:44:43.983 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T04:44:44.095 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-27T04:44:44.106 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-27T04:44:44.106 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T04:44:44.132 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-27T04:44:44.142 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T04:44:44.199 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2024-04-27T04:44:44.221 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T04:44:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T04:44:44.277 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-27T04:44:44.287 INFO:teuthology.task.internal:Transferring archived files... 2024-04-27T04:44:44.288 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-26_22:40:03-rgw-reef-distro-default-smithi/7675694/remote/smithi161 2024-04-27T04:44:44.288 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T04:44:44.356 INFO:teuthology.task.internal:Removing archive directory... 2024-04-27T04:44:44.356 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T04:44:44.410 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-27T04:44:44.422 INFO:teuthology.task.internal:Not uploading archives. 2024-04-27T04:44:44.422 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-27T04:44:44.433 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-27T04:44:44.433 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T04:44:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 04:44 /home/ubuntu/cephtest 2024-04-27T04:44:44.467 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-27T04:44:44.480 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-27T04:44:44.506 INFO:teuthology.run:Summary data: description: rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests} duration: 1845.1888332366943 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-27T04:44:44.507 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T04:44:44.593 INFO:teuthology.run:pass