2023-01-24T16:27:23.442 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-24T16:27:23.443 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135445 --description rgw/lifecycle/{cluster overrides supported-random-distro$/{rhel_8} tasks/rgw_s3tests} -- /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135445/orig.config.yaml 2023-01-24T16:27:23.469 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T16:27:23.524 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135445 branch: wip-yuri7-testing-2023-01-23-1532-quincy description: rgw/lifecycle/{cluster overrides supported-random-distro$/{rhel_8} tasks/rgw_s3tests} email: yweinstein@radhat.com first_in_suite: false job_id: '7135445' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri7-testing-2023-01-23-1532-quincy ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw lc debug interval: 10 setgroup: ceph setuser: ceph storage classes: LUKEWARM, FROZEN mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 49f8fb05584886826e8eade75f7105fba754560c wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 49f8fb05584886826e8eade75f7105fba754560c rgw: frontend: beast storage classes: LUKEWARM, FROZEN selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri7-testing-2023-01-23-1532-quincy sha1: 49f8fb05584886826e8eade75f7105fba754560c owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - osd.2 - mgr.0 - client.0 sha1: 49f8fb05584886826e8eade75f7105fba754560c sleep_before_teardown: 0 suite: rgw suite_branch: wip-yuri7-testing-2023-01-23-1532-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 49f8fb05584886826e8eade75f7105fba754560c targets: smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDH2P9SGj7EEEyNLxYTnBNzzYgzRuF3LWwwm4A9kPUSZVETbWpQyGzTXkCuC8JGJK6hlTXwbQxL5RzTNGiyXbtWfZ8owkv2lN9OCXtsbG07syuCg0IfNd6/dx67qXikCQYA/mW/2165tx0Lr1tzPApYcOIB6SqduX3cfs5jzos3o+HU6VP9UnmjBd3O6w2aw4RSdaHknpq9CjT9Qm5HuCr1hNlWu9p7OYOXkIYLK9LmNfVQSAA3Rz50THlCBRIlNQg1Boh1RfXChWiBFJ88jEKBm96E2R9oYQXb6FQmBujO5IzktunT/CSgheILfDyOX/Xj/h5j0ZOdJICPq+kr83mzM4dqPzV/b2bJbOWlntqi7JeJz7vZuHWHG3MWiek9RPFxn2FWO4Cv0VpDJ/Ttvq5F8C3TP1dEiMhUB7xpwk1eWSsb8tKAX1cyvHZXC8RPL/TM+qAlTYBR1qrTwNRVZLJf+ygsQ2wSeVw2ZCHbTmoL1q3ML2W+HHhVHifXVSMV5es= tasks: - install: null - ceph: null - rgw: - client.0 - s3tests: client.0: extra_attrs: - lifecycle force-branch: ceph-quincy lc_debug_interval: 10 rgw_server: client.0 storage classes: LUKEWARM, FROZEN teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_16:04:03 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-24T16:27:23.524 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa; will attempt to use it 2023-01-24T16:27:23.525 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa/tasks 2023-01-24T16:27:23.525 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-24T16:27:23.527 INFO:teuthology.task.internal:Checking packages... 2023-01-24T16:27:23.551 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '49f8fb05584886826e8eade75f7105fba754560c' 2023-01-24T16:27:23.552 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-24T16:27:23.552 INFO:teuthology.packaging:ref: None 2023-01-24T16:27:23.553 INFO:teuthology.packaging:tag: None 2023-01-24T16:27:23.553 INFO:teuthology.packaging:branch: wip-yuri7-testing-2023-01-23-1532-quincy 2023-01-24T16:27:23.553 INFO:teuthology.packaging:sha1: 49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T16:27:23.553 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri7-testing-2023-01-23-1532-quincy 2023-01-24T16:27:23.825 INFO:teuthology.task.internal:Found packages for ceph version 17.2.5-727.g49f8fb05 2023-01-24T16:27:23.825 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-24T16:27:23.835 INFO:teuthology.task.internal:no buildpackages task found 2023-01-24T16:27:23.835 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-24T16:27:23.856 INFO:teuthology.task.internal:Saving configuration 2023-01-24T16:27:23.867 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-24T16:27:23.875 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-24T16:27:23.896 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135445', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 16:22:25.301309', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA7L8SLLvGvTWh+g/1M1a3WH4ZfFSPrI3JxRvY0iruZIahBSCV5ZK/ToW+QeVVQhK32Q2+CYsW7vLRjrkGfVVQo='} 2023-01-24T16:27:23.897 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-24T16:27:23.903 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'osd.2', 'mgr.0', 'client.0'] 2023-01-24T16:27:23.904 INFO:teuthology.run_tasks:Running task console_log... 2023-01-24T16:27:23.942 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc395a72700>, signals=[15]) 2023-01-24T16:27:23.943 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-24T16:27:23.950 INFO:teuthology.task.internal:Opening connections... 2023-01-24T16:27:23.951 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2023-01-24T16:27:23.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T16:27:24.021 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-24T16:27:24.029 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2023-01-24T16:27:24.046 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2023-01-24T16:27:24.047 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2023-01-24T16:27:24.103 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="Red Hat Enterprise Linux" 2023-01-24T16:27:24.103 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8.4 (Ootpa)" 2023-01-24T16:27:24.103 INFO:teuthology.orchestra.run.smithi097.stdout:ID="rhel" 2023-01-24T16:27:24.103 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="fedora" 2023-01-24T16:27:24.104 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8.4" 2023-01-24T16:27:24.104 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8" 2023-01-24T16:27:24.104 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-01-24T16:27:24.104 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2023-01-24T16:27:24.104 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-01-24T16:27:24.105 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://www.redhat.com/" 2023-01-24T16:27:24.105 INFO:teuthology.orchestra.run.smithi097.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-01-24T16:27:24.105 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-24T16:27:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:27:24.105 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-24T16:27:24.106 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-01-24T16:27:24.106 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-01-24T16:27:24.106 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-01-24T16:27:24.107 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-01-24T16:27:24.121 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-24T16:27:24.131 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-24T16:27:24.139 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-24T16:27:24.140 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2023-01-24T16:27:24.158 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-24T16:27:24.166 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-24T16:27:24.167 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2023-01-24T16:27:24.213 INFO:teuthology.orchestra.run.smithi097.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-01-24T16:27:24.214 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-24T16:27:24.242 INFO:teuthology.run_tasks:Running task kernel... 2023-01-24T16:27:24.257 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-24T16:27:24.258 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-24T16:27:24.258 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-24T16:27:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T16:27:24.274 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2023-01-24T16:27:24.329 INFO:teuthology.orchestra.run.smithi097.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-01-24T16:27:24.329 INFO:teuthology.task.kernel:Running kernel on smithi097: 4.18.0-305.25.1.el8_4.x86_64 2023-01-24T16:27:24.329 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum install -y kernel 2023-01-24T16:27:25.786 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:27:25.787 INFO:teuthology.orchestra.run.smithi097.stdout:Unable to read consumer identity 2023-01-24T16:27:27.175 INFO:teuthology.orchestra.run.smithi097.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-01-24T16:27:30.272 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 13 MB 00:03 2023-01-24T16:27:33.045 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 3.9 MB/s | 24 kB 00:00 2023-01-24T16:27:33.735 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-01-24T16:27:33.735 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-01-24T16:27:33.735 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-01-24T16:27:33.859 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:27:33.860 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-01-24T16:27:33.860 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:27:33.969 DEBUG:teuthology.orchestra.run.smithi097:> echo no | sudo yum reinstall kernel || true 2023-01-24T16:27:34.311 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:27:34.311 INFO:teuthology.orchestra.run.smithi097.stdout:Unable to read consumer identity 2023-01-24T16:27:34.594 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:01 ago on Tue 24 Jan 2023 04:27:33 PM UTC. 2023-01-24T16:27:34.727 INFO:teuthology.orchestra.run.smithi097.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-01-24T16:27:34.727 INFO:teuthology.orchestra.run.smithi097.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-01-24T16:27:34.728 INFO:teuthology.orchestra.run.smithi097.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-01-24T16:27:34.759 INFO:teuthology.orchestra.run.smithi097.stderr:Error: No packages marked for reinstall. 2023-01-24T16:27:34.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum reinstall -y kernel || true 2023-01-24T16:27:35.148 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:27:35.149 INFO:teuthology.orchestra.run.smithi097.stdout:Unable to read consumer identity 2023-01-24T16:27:35.423 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Jan 2023 04:27:33 PM UTC. 2023-01-24T16:27:35.559 INFO:teuthology.orchestra.run.smithi097.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-01-24T16:27:35.560 INFO:teuthology.orchestra.run.smithi097.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-01-24T16:27:35.560 INFO:teuthology.orchestra.run.smithi097.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-01-24T16:27:35.569 INFO:teuthology.orchestra.run.smithi097.stderr:Error: No packages marked for reinstall. 2023-01-24T16:27:35.617 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel --last | head -n 1 2023-01-24T16:27:35.740 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-4.18.0-305.25.1.el8_4.x86_64 Wed 21 Dec 2022 01:05:24 PM UTC 2023-01-24T16:27:35.742 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.25.1.el8_4.x86_64 2023-01-24T16:27:35.742 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-01-24T16:27:35.742 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-24T16:27:35.742 DEBUG:teuthology.orchestra.run.smithi097:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-24T16:27:35.768 INFO:teuthology.orchestra.run.smithi097.stdout:ttyS1 2023-01-24T16:27:35.780 DEBUG:teuthology.parallel:result is None 2023-01-24T16:27:35.780 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-24T16:27:35.789 INFO:teuthology.task.internal:Creating test directory... 2023-01-24T16:27:35.789 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-24T16:27:35.834 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-24T16:27:35.845 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-24T16:27:35.859 INFO:teuthology.task.internal:Creating archive directory... 2023-01-24T16:27:35.860 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-24T16:27:35.897 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-24T16:27:35.908 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-24T16:27:35.908 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-24T16:27:35.970 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T16:27:35.979 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T16:27:35.981 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-24T16:27:35.989 INFO:teuthology.task.internal:Configuring sudo... 2023-01-24T16:27:35.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-24T16:27:36.045 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-24T16:27:36.057 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-24T16:27:36.057 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-24T16:27:36.099 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T16:27:36.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T16:27:36.219 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T16:27:36.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T16:27:36.337 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:27:36.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-24T16:27:36.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2023-01-24T16:27:36.481 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T16:27:36.497 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-24T16:27:36.508 INFO:teuthology.task.internal:Starting timer... 2023-01-24T16:27:36.509 INFO:teuthology.run_tasks:Running task pcp... 2023-01-24T16:27:36.526 INFO:teuthology.run_tasks:Running task selinux... 2023-01-24T16:27:36.540 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-01-24T16:27:36.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2023-01-24T16:27:36.595 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: [ OK ] 2023-01-24T16:27:36.596 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-24T16:27:36.597 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2023-01-24T16:27:36.618 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2023-01-24T16:27:36.619 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi097.front.sepia.ceph.com': 'permissive'} 2023-01-24T16:27:36.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T16:27:36.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T16:27:36.683 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-24T16:27:36.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2023-01-24T16:27:36.746 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-24T16:27:36.760 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-01-24T16:27:36.761 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-24T16:27:36.776 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-24T16:27:36.777 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi097.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-24T16:33:56.823 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi097.front.sepia.ceph.com')] 2023-01-24T16:33:56.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-24T16:33:56.826 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T16:33:56.895 DEBUG:teuthology.orchestra.run.smithi097:> true 2023-01-24T16:33:56.958 INFO:teuthology.run_tasks:Running task clock... 2023-01-24T16:33:56.969 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-24T16:33:56.970 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-24T16:33:56.971 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T16:33:57.027 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-24T16:33:57.040 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-24T16:33:57.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2023-01-24T16:33:57.088 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2023-01-24T16:33:57.101 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-24T16:33:57.114 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-24T16:33:57.179 INFO:teuthology.orchestra.run.smithi097.stderr:bash: ntpq: command not found 2023-01-24T16:33:57.182 INFO:teuthology.orchestra.run.smithi097.stdout:210 Number of sources = 4 2023-01-24T16:33:57.182 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T16:33:57.182 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2023-01-24T16:33:57.183 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T16:33:57.183 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T16:33:57.183 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T16:33:57.183 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T16:33:57.183 INFO:teuthology.run_tasks:Running task install... 2023-01-24T16:33:57.192 DEBUG:teuthology.task.install:project ceph 2023-01-24T16:33:57.193 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}} 2023-01-24T16:33:57.193 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'} 2023-01-24T16:33:57.193 INFO:teuthology.task.install:Using flavor: default 2023-01-24T16:33:57.200 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-24T16:33:57.200 INFO:teuthology.task.install:extra packages: [] 2023-01-24T16:33:57.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo lsb_release -is 2023-01-24T16:33:57.259 INFO:teuthology.orchestra.run.smithi097.stdout:RedHatEnterprise 2023-01-24T16:33:57.260 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2023-01-24T16:33:57.260 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'tag': None, 'wait_for_package': False} 2023-01-24T16:33:57.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo lsb_release -is 2023-01-24T16:33:57.336 INFO:teuthology.orchestra.run.smithi097.stdout:RedHatEnterprise 2023-01-24T16:33:57.337 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2023-01-24T16:33:57.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T16:33:57.492 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/centos/8/flavors/default/ 2023-01-24T16:33:57.493 INFO:teuthology.task.install.rpm:Package version is 17.2.5-727.g49f8fb05 2023-01-24T16:33:57.616 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-24T16:33:57.617 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:33:57.617 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-24T16:33:57.642 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-01-24T16:33:57.643 DEBUG:teuthology.orchestra.run.smithi097:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/49f8fb05584886826e8eade75f7105fba754560c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-24T16:33:57.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-24T16:33:57.782 DEBUG:teuthology.orchestra.run.smithi097:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-24T16:33:57.844 INFO:teuthology.orchestra.run.smithi097.stdout:check_obsoletes = 1 2023-01-24T16:33:57.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum clean all 2023-01-24T16:33:58.232 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:33:58.529 INFO:teuthology.orchestra.run.smithi097.stdout:44 files removed 2023-01-24T16:33:58.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw 2023-01-24T16:33:58.921 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:33:59.577 INFO:teuthology.orchestra.run.smithi097.stdout:ceph packages for x86_64 323 kB/s | 78 kB 00:00 2023-01-24T16:33:59.820 INFO:teuthology.orchestra.run.smithi097.stdout:ceph noarch packages 46 kB/s | 9.2 kB 00:00 2023-01-24T16:34:00.054 INFO:teuthology.orchestra.run.smithi097.stdout:ceph source packages 7.2 kB/s | 1.5 kB 00:00 2023-01-24T16:34:00.155 INFO:teuthology.orchestra.run.smithi097.stdout:Copr repo for python3-asyncssh owned by ceph 48 kB/s | 3.5 kB 00:00 2023-01-24T16:34:01.026 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 16 MB/s | 13 MB 00:00 2023-01-24T16:34:03.875 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2023-01-24T16:34:04.237 INFO:teuthology.orchestra.run.smithi097.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 134 MB/s | 38 MB 00:00 2023-01-24T16:34:09.747 INFO:teuthology.orchestra.run.smithi097.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 125 MB/s | 33 MB 00:00 2023-01-24T16:34:18.101 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:34:18.104 INFO:teuthology.orchestra.run.smithi097.stdout:=================================================================================================== 2023-01-24T16:34:18.104 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-01-24T16:34:18.104 INFO:teuthology.orchestra.run.smithi097.stdout:=================================================================================================== 2023-01-24T16:34:18.104 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:34:18.105 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 17 M 2023-01-24T16:34:18.105 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2023-01-24T16:34:18.105 INFO:teuthology.orchestra.run.smithi097.stdout: librados2 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 3.5 M 2023-01-24T16:34:18.105 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 3.6 M 2023-01-24T16:34:18.105 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-01-24T16:34:18.106 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 5.1 M 2023-01-24T16:34:18.106 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 23 M 2023-01-24T16:34:18.106 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 24 k 2023-01-24T16:34:18.106 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-24T16:34:18.106 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 793 k 2023-01-24T16:34:18.107 INFO:teuthology.orchestra.run.smithi097.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-24T16:34:18.107 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-01-24T16:34:18.107 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 506 k 2023-01-24T16:34:18.107 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-01-24T16:34:18.107 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 5.5 M 2023-01-24T16:34:18.108 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-24T16:34:18.108 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-01-24T16:34:18.108 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 45 k 2023-01-24T16:34:18.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 108 k 2023-01-24T16:34:18.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 190 k 2023-01-24T16:34:18.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-01-24T16:34:18.109 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 370 k 2023-01-24T16:34:18.110 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 357 k 2023-01-24T16:34:18.110 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 107 k 2023-01-24T16:34:18.110 INFO:teuthology.orchestra.run.smithi097.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-24T16:34:18.110 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:34:18.110 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:34:18.111 INFO:teuthology.orchestra.run.smithi097.stdout:=================================================================================================== 2023-01-24T16:34:18.111 INFO:teuthology.orchestra.run.smithi097.stdout:Install 21 Packages 2023-01-24T16:34:18.111 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 2 Packages 2023-01-24T16:34:18.111 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:34:18.112 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 63 M 2023-01-24T16:34:18.112 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:34:18.655 INFO:teuthology.orchestra.run.smithi097.stdout:(1/23): ceph-base-17.2.5-727.g49f8fb05.el8.x86_ 9.4 MB/s | 5.1 MB 00:00 2023-01-24T16:34:18.697 INFO:teuthology.orchestra.run.smithi097.stdout:(2/23): ceph-selinux-17.2.5-727.g49f8fb05.el8.x 577 kB/s | 24 kB 00:00 2023-01-24T16:34:18.771 INFO:teuthology.orchestra.run.smithi097.stdout:(3/23): libcephfs2-17.2.5-727.g49f8fb05.el8.x86 11 MB/s | 793 kB 00:00 2023-01-24T16:34:18.838 INFO:teuthology.orchestra.run.smithi097.stdout:(4/23): libradosstriper1-17.2.5-727.g49f8fb05.e 7.4 MB/s | 506 kB 00:00 2023-01-24T16:34:19.313 INFO:teuthology.orchestra.run.smithi097.stdout:(5/23): librgw2-17.2.5-727.g49f8fb05.el8.x86_64 12 MB/s | 5.5 MB 00:00 2023-01-24T16:34:19.352 INFO:teuthology.orchestra.run.smithi097.stdout:(6/23): python3-ceph-argparse-17.2.5-727.g49f8f 1.1 MB/s | 45 kB 00:00 2023-01-24T16:34:19.393 INFO:teuthology.orchestra.run.smithi097.stdout:(7/23): python3-ceph-common-17.2.5-727.g49f8fb0 2.6 MB/s | 108 kB 00:00 2023-01-24T16:34:19.439 INFO:teuthology.orchestra.run.smithi097.stdout:(8/23): python3-cephfs-17.2.5-727.g49f8fb05.el8 4.2 MB/s | 190 kB 00:00 2023-01-24T16:34:19.489 INFO:teuthology.orchestra.run.smithi097.stdout:(9/23): python3-rados-17.2.5-727.g49f8fb05.el8. 7.1 MB/s | 370 kB 00:00 2023-01-24T16:34:19.540 INFO:teuthology.orchestra.run.smithi097.stdout:(10/23): python3-rbd-17.2.5-727.g49f8fb05.el8.x 6.9 MB/s | 357 kB 00:00 2023-01-24T16:34:19.582 INFO:teuthology.orchestra.run.smithi097.stdout:(11/23): python3-rgw-17.2.5-727.g49f8fb05.el8.x 2.5 MB/s | 107 kB 00:00 2023-01-24T16:34:19.747 INFO:teuthology.orchestra.run.smithi097.stdout:(12/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2023-01-24T16:34:19.775 INFO:teuthology.orchestra.run.smithi097.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 2.1 MB/s | 59 kB 00:00 2023-01-24T16:34:19.804 INFO:teuthology.orchestra.run.smithi097.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.6 MB/s | 75 kB 00:00 2023-01-24T16:34:19.965 INFO:teuthology.orchestra.run.smithi097.stdout:(15/23): ceph-radosgw-17.2.5-727.g49f8fb05.el8. 9.0 MB/s | 17 MB 00:01 2023-01-24T16:34:19.996 INFO:teuthology.orchestra.run.smithi097.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.5 MB/s | 47 kB 00:00 2023-01-24T16:34:20.010 INFO:teuthology.orchestra.run.smithi097.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 3.1 MB/s | 44 kB 00:00 2023-01-24T16:34:20.027 INFO:teuthology.orchestra.run.smithi097.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 16 MB/s | 259 kB 00:00 2023-01-24T16:34:20.043 INFO:teuthology.orchestra.run.smithi097.stdout:(19/23): librdkafka-0.11.4-1.el8.x86_64.rpm 21 MB/s | 353 kB 00:00 2023-01-24T16:34:20.052 INFO:teuthology.orchestra.run.smithi097.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 7.0 MB/s | 1.7 MB 00:00 2023-01-24T16:34:20.378 INFO:teuthology.orchestra.run.smithi097.stdout:(21/23): librados2-17.2.5-727.g49f8fb05.el8.x86 10 MB/s | 3.5 MB 00:00 2023-01-24T16:34:20.433 INFO:teuthology.orchestra.run.smithi097.stdout:(22/23): librbd1-17.2.5-727.g49f8fb05.el8.x86_6 9.4 MB/s | 3.6 MB 00:00 2023-01-24T16:34:20.553 INFO:teuthology.orchestra.run.smithi097.stdout:(23/23): ceph-common-17.2.5-727.g49f8fb05.el8.x 9.5 MB/s | 23 MB 00:02 2023-01-24T16:34:20.556 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:34:20.556 INFO:teuthology.orchestra.run.smithi097.stdout:Total 26 MB/s | 63 MB 00:02 2023-01-24T16:34:20.560 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:34:20.603 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:34:20.604 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:34:21.405 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:34:21.409 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:34:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:34:23.120 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-01-24T16:34:23.488 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-01-24T16:34:23.709 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 2/25 2023-01-24T16:34:23.804 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 2/25 2023-01-24T16:34:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-2:17.2.5-727.g49f8fb05.el8.x86_64 3/25 2023-01-24T16:34:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-01-24T16:34:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 5/25 2023-01-24T16:34:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 5/25 2023-01-24T16:34:25.159 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 6/25 2023-01-24T16:34:25.256 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 6/25 2023-01-24T16:34:25.492 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/25 2023-01-24T16:34:25.565 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/25 2023-01-24T16:34:25.924 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-01-24T16:34:26.624 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-01-24T16:34:26.825 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librgw2-2:17.2.5-727.g49f8fb05.el8.x86_64 10/25 2023-01-24T16:34:26.875 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librgw2-2:17.2.5-727.g49f8fb05.el8.x86_64 10/25 2023-01-24T16:34:27.010 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-argparse-2:17.2.5-727.g49f8fb05.el8.x 11/25 2023-01-24T16:34:27.237 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-2:17.2.5-727.g49f8fb05.el8.x86_64 12/25 2023-01-24T16:34:27.477 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-2:17.2.5-727.g49f8fb05.el8.x86_64 13/25 2023-01-24T16:34:27.720 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-2:17.2.5-727.g49f8fb05.el8.x86_64 14/25 2023-01-24T16:34:27.916 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libradosstriper1-2:17.2.5-727.g49f8fb05.el8.x86_64 15/25 2023-01-24T16:34:27.988 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libradosstriper1-2:17.2.5-727.g49f8fb05.el8.x86_64 15/25 2023-01-24T16:34:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-01-24T16:34:28.393 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-01-24T16:34:28.679 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-01-24T16:34:28.842 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-common-2:17.2.5-727.g49f8fb05.el8.x86 19/25 2023-01-24T16:34:33.576 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 20/25 2023-01-24T16:34:33.783 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 20/25 2023-01-24T16:34:34.345 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 20/25 2023-01-24T16:34:34.668 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 21/25 2023-01-24T16:34:34.758 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 21/25 2023-01-24T16:34:34.874 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-selinux-2:17.2.5-727.g49f8fb05.el8.x86_64 22/25 2023-01-24T16:34:52.740 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-selinux-2:17.2.5-727.g49f8fb05.el8.x86_64 22/25 2023-01-24T16:34:52.740 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /sys 2023-01-24T16:34:52.740 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /proc 2023-01-24T16:34:52.741 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /dev 2023-01-24T16:34:52.742 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /run 2023-01-24T16:34:52.742 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /mnt 2023-01-24T16:34:52.742 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /var/tmp 2023-01-24T16:34:52.742 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /home 2023-01-24T16:34:52.742 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /tmp 2023-01-24T16:34:52.743 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /dev 2023-01-24T16:34:52.743 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:34:54.482 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-2:17.2.5-727.g49f8fb05.el8.x86_64 23/25 2023-01-24T16:34:54.557 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-2:17.2.5-727.g49f8fb05.el8.x86_64 23/25 2023-01-24T16:34:54.558 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T16:34:54.757 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T16:34:54.758 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T16:34:55.318 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T16:34:55.318 INFO:teuthology.orchestra.run.smithi097.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-01-24T16:34:55.318 INFO:teuthology.orchestra.run.smithi097.stdout:All rules containing unresolvable specifiers will be skipped. 2023-01-24T16:34:55.319 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:34:55.596 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 1/25 2023-01-24T16:34:55.596 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 2/25 2023-01-24T16:34:55.596 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-2:17.2.5-727.g49f8fb05.el8.x86_64 3/25 2023-01-24T16:34:55.597 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-selinux-2:17.2.5-727.g49f8fb05.el8.x86_64 4/25 2023-01-24T16:34:55.597 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 5/25 2023-01-24T16:34:55.597 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libradosstriper1-2:17.2.5-727.g49f8fb05.el8.x86_64 6/25 2023-01-24T16:34:55.597 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librgw2-2:17.2.5-727.g49f8fb05.el8.x86_64 7/25 2023-01-24T16:34:55.597 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-argparse-2:17.2.5-727.g49f8fb05.el8.x 8/25 2023-01-24T16:34:55.598 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-common-2:17.2.5-727.g49f8fb05.el8.x86 9/25 2023-01-24T16:34:55.598 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-2:17.2.5-727.g49f8fb05.el8.x86_64 10/25 2023-01-24T16:34:55.598 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-2:17.2.5-727.g49f8fb05.el8.x86_64 11/25 2023-01-24T16:34:55.598 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-2:17.2.5-727.g49f8fb05.el8.x86_64 12/25 2023-01-24T16:34:55.598 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-2:17.2.5-727.g49f8fb05.el8.x86_64 13/25 2023-01-24T16:34:55.599 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-01-24T16:34:55.599 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-01-24T16:34:55.599 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-01-24T16:34:55.599 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-01-24T16:34:55.599 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-01-24T16:34:55.600 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2023-01-24T16:34:55.600 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2023-01-24T16:34:55.600 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/25 2023-01-24T16:34:55.600 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 22/25 2023-01-24T16:34:55.600 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-01-24T16:34:55.606 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 24/25 2023-01-24T16:34:56.975 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T16:34:56.975 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:47 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:34:58.894 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:34:59.000 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:34:59.001 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2023-01-24T16:34:59.001 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.001 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.002 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:34:59.002 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.002 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.003 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.003 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-24T16:34:59.004 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-24T16:34:59.005 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-24T16:34:59.006 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.006 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-01-24T16:34:59.006 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.007 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-24T16:34:59.007 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-24T16:34:59.007 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.007 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.008 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.008 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-24T16:34:59.008 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.009 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.009 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:34:59.010 INFO:teuthology.orchestra.run.smithi097.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-24T16:34:59.010 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:34:59.010 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:34:59.165 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-test 2023-01-24T16:34:59.523 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:35:00.311 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:51 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:35:02.083 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:35:02.083 INFO:teuthology.orchestra.run.smithi097.stdout:=========================================================================================== 2023-01-24T16:35:02.084 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-01-24T16:35:02.084 INFO:teuthology.orchestra.run.smithi097.stdout:=========================================================================================== 2023-01-24T16:35:02.084 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:35:02.084 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 52 M 2023-01-24T16:35:02.085 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-01-24T16:35:02.085 INFO:teuthology.orchestra.run.smithi097.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-01-24T16:35:02.085 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 174 k 2023-01-24T16:35:02.085 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-01-24T16:35:02.086 INFO:teuthology.orchestra.run.smithi097.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-01-24T16:35:02.086 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-01-24T16:35:02.086 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:02.087 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:35:02.087 INFO:teuthology.orchestra.run.smithi097.stdout:=========================================================================================== 2023-01-24T16:35:02.087 INFO:teuthology.orchestra.run.smithi097.stdout:Install 6 Packages 2023-01-24T16:35:02.088 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:02.088 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 53 M 2023-01-24T16:35:02.089 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 233 M 2023-01-24T16:35:02.089 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:35:02.089 INFO:teuthology.orchestra.run.smithi097.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 19 MB/s | 70 kB 00:00 2023-01-24T16:35:02.120 INFO:teuthology.orchestra.run.smithi097.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 5.2 MB/s | 161 kB 00:00 2023-01-24T16:35:02.136 INFO:teuthology.orchestra.run.smithi097.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 14 MB/s | 187 kB 00:00 2023-01-24T16:35:02.149 INFO:teuthology.orchestra.run.smithi097.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 22 MB/s | 302 kB 00:00 2023-01-24T16:35:02.259 INFO:teuthology.orchestra.run.smithi097.stdout:(5/6): libcephsqlite-17.2.5-727.g49f8fb05.el8.x 1.0 MB/s | 174 kB 00:00 2023-01-24T16:35:04.448 INFO:teuthology.orchestra.run.smithi097.stdout:(6/6): ceph-test-17.2.5-727.g49f8fb05.el8.x86_6 22 MB/s | 52 MB 00:02 2023-01-24T16:35:04.451 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:35:04.452 INFO:teuthology.orchestra.run.smithi097.stdout:Total 22 MB/s | 53 MB 00:02 2023-01-24T16:35:04.455 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:35:04.479 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:35:04.480 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:35:04.850 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:35:04.854 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:35:05.360 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:35:05.604 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-01-24T16:35:05.803 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-01-24T16:35:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-01-24T16:35:06.082 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-01-24T16:35:06.293 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-01-24T16:35:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephsqlite-2:17.2.5-727.g49f8fb05.el8.x86_64 5/6 2023-01-24T16:35:11.437 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephsqlite-2:17.2.5-727.g49f8fb05.el8.x86_64 5/6 2023-01-24T16:35:11.709 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-2:17.2.5-727.g49f8fb05.el8.x86_64 6/6 2023-01-24T16:35:11.972 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-test-2:17.2.5-727.g49f8fb05.el8.x86_64 6/6 2023-01-24T16:35:11.972 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-2:17.2.5-727.g49f8fb05.el8.x86_64 1/6 2023-01-24T16:35:11.973 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephsqlite-2:17.2.5-727.g49f8fb05.el8.x86_64 2/6 2023-01-24T16:35:11.973 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-01-24T16:35:11.973 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-01-24T16:35:11.973 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-01-24T16:35:12.926 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-01-24T16:35:12.927 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:03 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:35:14.767 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:35:14.864 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:14.864 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:35:14.865 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:35:14.865 INFO:teuthology.orchestra.run.smithi097.stdout: jq-1.5-12.el8.x86_64 2023-01-24T16:35:14.865 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:35:14.865 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-24T16:35:14.865 INFO:teuthology.orchestra.run.smithi097.stdout: socat-1.7.3.3-2.el8.x86_64 2023-01-24T16:35:14.866 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-24T16:35:14.866 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:14.866 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:35:14.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph 2023-01-24T16:35:15.324 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:35:16.053 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:07 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:35:17.802 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:35:17.804 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================================== 2023-01-24T16:35:17.804 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-01-24T16:35:17.805 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================================== 2023-01-24T16:35:17.805 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:35:17.805 INFO:teuthology.orchestra.run.smithi097.stdout: ceph x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 6.4 k 2023-01-24T16:35:17.805 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-01-24T16:35:17.805 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 2.3 M 2023-01-24T16:35:17.806 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 1.6 M 2023-01-24T16:35:17.806 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 233 k 2023-01-24T16:35:17.806 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 3.9 M 2023-01-24T16:35:17.806 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 17 M 2023-01-24T16:35:17.806 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-24T16:35:17.807 INFO:teuthology.orchestra.run.smithi097.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-24T16:35:17.807 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-24T16:35:17.807 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-24T16:35:17.807 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-24T16:35:17.807 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-24T16:35:17.807 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-24T16:35:17.808 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-01-24T16:35:17.808 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-01-24T16:35:17.808 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-24T16:35:17.808 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-24T16:35:17.809 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-24T16:35:17.809 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-01-24T16:35:17.809 INFO:teuthology.orchestra.run.smithi097.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-24T16:35:17.809 INFO:teuthology.orchestra.run.smithi097.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-24T16:35:17.809 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-24T16:35:17.809 INFO:teuthology.orchestra.run.smithi097.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-24T16:35:17.810 INFO:teuthology.orchestra.run.smithi097.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-24T16:35:17.810 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-24T16:35:17.810 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-24T16:35:17.810 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-01-24T16:35:17.810 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-24T16:35:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:17.811 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:35:17.811 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================================== 2023-01-24T16:35:17.811 INFO:teuthology.orchestra.run.smithi097.stdout:Install 28 Packages 2023-01-24T16:35:17.811 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:17.812 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 27 M 2023-01-24T16:35:17.812 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 105 M 2023-01-24T16:35:17.813 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:35:17.909 INFO:teuthology.orchestra.run.smithi097.stdout:(1/28): ceph-17.2.5-727.g49f8fb05.el8.x86_64.rp 64 kB/s | 6.4 kB 00:00 2023-01-24T16:35:18.165 INFO:teuthology.orchestra.run.smithi097.stdout:(2/28): ceph-mds-17.2.5-727.g49f8fb05.el8.x86_6 6.4 MB/s | 2.3 MB 00:00 2023-01-24T16:35:18.169 INFO:teuthology.orchestra.run.smithi097.stdout:(3/28): ceph-mgr-17.2.5-727.g49f8fb05.el8.x86_6 4.3 MB/s | 1.6 MB 00:00 2023-01-24T16:35:18.237 INFO:teuthology.orchestra.run.smithi097.stdout:(4/28): ceph-mgr-modules-core-17.2.5-727.g49f8f 3.4 MB/s | 233 kB 00:00 2023-01-24T16:35:18.326 INFO:teuthology.orchestra.run.smithi097.stdout:(5/28): ceph-mon-17.2.5-727.g49f8fb05.el8.x86_6 9.4 MB/s | 3.9 MB 00:00 2023-01-24T16:35:18.348 INFO:teuthology.orchestra.run.smithi097.stdout:(6/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 396 kB/s | 44 kB 00:00 2023-01-24T16:35:18.430 INFO:teuthology.orchestra.run.smithi097.stdout:(7/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.1 MB/s | 173 kB 00:00 2023-01-24T16:35:18.468 INFO:teuthology.orchestra.run.smithi097.stdout:(8/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 1.3 MB/s | 185 kB 00:00 2023-01-24T16:35:18.487 INFO:teuthology.orchestra.run.smithi097.stdout:(9/28): python3-cherrypy-18.4.0-1.el8.noarch.rp 6.6 MB/s | 384 kB 00:00 2023-01-24T16:35:18.496 INFO:teuthology.orchestra.run.smithi097.stdout:(10/28): python3-jaraco-6.2-6.el8.noarch.rpm 394 kB/s | 11 kB 00:00 2023-01-24T16:35:18.515 INFO:teuthology.orchestra.run.smithi097.stdout:(11/28): python3-jaraco-functools-2.0-4.el8.noa 630 kB/s | 18 kB 00:00 2023-01-24T16:35:18.525 INFO:teuthology.orchestra.run.smithi097.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 1.7 MB/s | 49 kB 00:00 2023-01-24T16:35:18.543 INFO:teuthology.orchestra.run.smithi097.stdout:(13/28): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2023-01-24T16:35:18.571 INFO:teuthology.orchestra.run.smithi097.stdout:(14/28): python3-portend-2.6-1.el8.noarch.rpm 567 kB/s | 16 kB 00:00 2023-01-24T16:35:18.582 INFO:teuthology.orchestra.run.smithi097.stdout:(15/28): python3-pecan-1.3.2-9.el8.noarch.rpm 4.8 MB/s | 283 kB 00:00 2023-01-24T16:35:18.599 INFO:teuthology.orchestra.run.smithi097.stdout:(16/28): python3-simplegeneric-0.8.1-17.el8.noa 668 kB/s | 19 kB 00:00 2023-01-24T16:35:18.611 INFO:teuthology.orchestra.run.smithi097.stdout:(17/28): python3-singledispatch-3.4.0.3-18.el8. 841 kB/s | 24 kB 00:00 2023-01-24T16:35:18.629 INFO:teuthology.orchestra.run.smithi097.stdout:(18/28): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2023-01-24T16:35:18.640 INFO:teuthology.orchestra.run.smithi097.stdout:(19/28): python3-trustme-0.6.0-4.el8.noarch.rpm 962 kB/s | 27 kB 00:00 2023-01-24T16:35:18.659 INFO:teuthology.orchestra.run.smithi097.stdout:(20/28): python3-waitress-1.2.1-2.el8.1.noarch. 7.7 MB/s | 241 kB 00:00 2023-01-24T16:35:18.671 INFO:teuthology.orchestra.run.smithi097.stdout:(21/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 8.0 MB/s | 251 kB 00:00 2023-01-24T16:35:18.687 INFO:teuthology.orchestra.run.smithi097.stdout:(22/28): python3-webtest-2.0.33-1.el8.noarch.rp 2.9 MB/s | 85 kB 00:00 2023-01-24T16:35:18.702 INFO:teuthology.orchestra.run.smithi097.stdout:(23/28): python3-zc-lockfile-2.0-2.el8.noarch.r 742 kB/s | 23 kB 00:00 2023-01-24T16:35:18.724 INFO:teuthology.orchestra.run.smithi097.stdout:(24/28): python3-werkzeug-0.12.2-4.el8.noarch.r 12 MB/s | 457 kB 00:00 2023-01-24T16:35:18.734 INFO:teuthology.orchestra.run.smithi097.stdout:(25/28): python3-markupsafe-0.23-19.el8.x86_64. 1.2 MB/s | 39 kB 00:00 2023-01-24T16:35:18.741 INFO:teuthology.orchestra.run.smithi097.stdout:(26/28): python3-mako-1.0.6-13.el8.noarch.rpm 9.2 MB/s | 157 kB 00:00 2023-01-24T16:35:18.747 INFO:teuthology.orchestra.run.smithi097.stdout:(27/28): python3-pytz-2017.2-9.el8.noarch.rpm 3.7 MB/s | 54 kB 00:00 2023-01-24T16:35:18.947 INFO:teuthology.orchestra.run.smithi097.stdout:(28/28): ceph-osd-17.2.5-727.g49f8fb05.el8.x86_ 21 MB/s | 17 MB 00:00 2023-01-24T16:35:18.950 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:35:18.951 INFO:teuthology.orchestra.run.smithi097.stdout:Total 24 MB/s | 27 MB 00:01 2023-01-24T16:35:18.954 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:35:18.990 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:35:18.990 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:35:19.372 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:35:19.376 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:35:20.436 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:35:20.642 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-01-24T16:35:20.899 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-01-24T16:35:21.055 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-01-24T16:35:21.331 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-01-24T16:35:21.691 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-01-24T16:35:21.935 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-01-24T16:35:22.160 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-01-24T16:35:22.392 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 8/28 2023-01-24T16:35:22.558 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-01-24T16:35:22.839 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-01-24T16:35:23.056 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-01-24T16:35:23.246 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-01-24T16:35:23.417 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-01-24T16:35:23.645 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-01-24T16:35:23.857 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-01-24T16:35:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-01-24T16:35:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-01-24T16:35:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-01-24T16:35:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-01-24T16:35:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-01-24T16:35:25.310 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-01-24T16:35:25.644 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-01-24T16:35:26.015 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-modules-core-2:17.2.5-727.g49f8fb05.el8.n 23/28 2023-01-24T16:35:26.228 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 24/28 2023-01-24T16:35:27.710 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 24/28 2023-01-24T16:35:28.658 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-2:17.2.5-727.g49f8fb05.el8.x86_64 25/28 2023-01-24T16:35:29.054 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-2:17.2.5-727.g49f8fb05.el8.x86_64 25/28 2023-01-24T16:35:29.309 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-2:17.2.5-727.g49f8fb05.el8.x86_64 26/28 2023-01-24T16:35:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-2:17.2.5-727.g49f8fb05.el8.x86_64 26/28 2023-01-24T16:35:29.683 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-2:17.2.5-727.g49f8fb05.el8.x86_64 27/28 2023-01-24T16:35:29.701 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-2:17.2.5-727.g49f8fb05.el8.x86_64 27/28 2023-01-24T16:35:29.858 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-2:17.2.5-727.g49f8fb05.el8.x86_64 28/28 2023-01-24T16:35:30.248 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-2:17.2.5-727.g49f8fb05.el8.x86_64 28/28 2023-01-24T16:35:30.248 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-2:17.2.5-727.g49f8fb05.el8.x86_64 1/28 2023-01-24T16:35:30.248 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-2:17.2.5-727.g49f8fb05.el8.x86_64 2/28 2023-01-24T16:35:30.249 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 3/28 2023-01-24T16:35:30.249 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-2:17.2.5-727.g49f8fb05.el8.x86_64 4/28 2023-01-24T16:35:30.249 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-2:17.2.5-727.g49f8fb05.el8.x86_64 5/28 2023-01-24T16:35:30.250 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-modules-core-2:17.2.5-727.g49f8fb05.el8.n 6/28 2023-01-24T16:35:30.250 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-01-24T16:35:30.251 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-01-24T16:35:30.251 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-01-24T16:35:30.251 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-01-24T16:35:30.251 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-01-24T16:35:30.252 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-01-24T16:35:30.252 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-01-24T16:35:30.252 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-01-24T16:35:30.252 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-01-24T16:35:30.252 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-01-24T16:35:30.253 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-01-24T16:35:30.253 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-01-24T16:35:30.253 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-01-24T16:35:30.253 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-01-24T16:35:30.253 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 21/28 2023-01-24T16:35:30.254 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-01-24T16:35:30.254 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-01-24T16:35:30.254 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-01-24T16:35:30.254 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-01-24T16:35:30.255 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-01-24T16:35:30.255 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-01-24T16:35:31.785 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-01-24T16:35:31.786 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:22 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:35:33.582 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:35:33.680 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:33.681 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:35:33.681 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:35:33.681 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:35:33.681 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:35:33.682 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T16:35:33.682 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:35:33.682 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:35:33.682 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-24T16:35:33.682 INFO:teuthology.orchestra.run.smithi097.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-24T16:35:33.683 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-24T16:35:33.683 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-24T16:35:33.683 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-24T16:35:33.683 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-24T16:35:33.683 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-24T16:35:33.684 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-24T16:35:33.686 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-24T16:35:33.686 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-24T16:35:33.686 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-24T16:35:33.686 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend-2.6-1.el8.noarch 2023-01-24T16:35:33.687 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-24T16:35:33.687 INFO:teuthology.orchestra.run.smithi097.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-24T16:35:33.687 INFO:teuthology.orchestra.run.smithi097.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-24T16:35:33.687 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-24T16:35:33.687 INFO:teuthology.orchestra.run.smithi097.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-24T16:35:33.688 INFO:teuthology.orchestra.run.smithi097.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-24T16:35:33.688 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-24T16:35:33.688 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-24T16:35:33.688 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-24T16:35:33.688 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-24T16:35:33.689 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:33.689 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:35:33.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-base 2023-01-24T16:35:34.206 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:35:34.955 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:25 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:35:36.308 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T16:35:36.761 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:35:36.762 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-01-24T16:35:36.762 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:35:36.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install cephadm 2023-01-24T16:35:37.211 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:35:37.940 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:28 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:35:40.534 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:35:40.536 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:35:40.536 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-01-24T16:35:40.537 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:35:40.537 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:35:40.537 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 78 k 2023-01-24T16:35:40.538 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:40.538 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:35:40.539 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:35:40.539 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-01-24T16:35:40.540 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:40.540 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 78 k 2023-01-24T16:35:40.540 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 355 k 2023-01-24T16:35:40.540 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:35:40.788 INFO:teuthology.orchestra.run.smithi097.stdout:cephadm-17.2.5-727.g49f8fb05.el8.noarch.rpm 310 kB/s | 78 kB 00:00 2023-01-24T16:35:40.789 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:35:40.789 INFO:teuthology.orchestra.run.smithi097.stdout:Total 308 kB/s | 78 kB 00:00 2023-01-24T16:35:40.790 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:35:40.798 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:35:40.798 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:35:40.800 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:35:40.804 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:35:40.950 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:35:43.388 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T16:35:43.602 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T16:35:43.800 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T16:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T16:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:35 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:35:46.647 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:35:46.746 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:46.746 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:35:46.746 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T16:35:46.746 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:46.747 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:35:46.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-immutable-object-cache 2023-01-24T16:35:47.243 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:35:48.028 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:39 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:35:49.799 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:35:49.800 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:35:49.800 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-01-24T16:35:49.801 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:35:49.801 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:35:49.801 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 162 k 2023-01-24T16:35:49.801 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:49.802 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:35:49.802 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:35:49.802 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-01-24T16:35:49.802 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:49.802 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 162 k 2023-01-24T16:35:49.803 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 490 k 2023-01-24T16:35:49.803 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:35:49.991 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-immutable-object-cache-17.2.5-727.g49f8fb0 851 kB/s | 162 kB 00:00 2023-01-24T16:35:49.992 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:35:49.992 INFO:teuthology.orchestra.run.smithi097.stdout:Total 844 kB/s | 162 kB 00:00 2023-01-24T16:35:49.993 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:35:50.005 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:35:50.005 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:35:50.057 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:35:50.062 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:35:50.177 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:35:50.397 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-2:17.2.5-727.g49f8fb05.e 1/1 2023-01-24T16:35:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.5-727.g49f8fb05.e 1/1 2023-01-24T16:35:51.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-2:17.2.5-727.g49f8fb05.e 1/1 2023-01-24T16:35:51.731 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:42 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:35:53.604 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:35:53.698 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:53.699 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:35:53.699 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:35:53.699 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:35:53.699 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:35:53.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr 2023-01-24T16:35:54.167 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:35:54.893 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:45 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:35:56.205 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T16:35:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:35:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-01-24T16:35:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:35:56.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-dashboard 2023-01-24T16:35:57.339 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:35:58.232 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:49 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:36:00.044 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:36:00.045 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================================== 2023-01-24T16:36:00.045 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-01-24T16:36:00.045 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================================== 2023-01-24T16:36:00.046 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:36:00.046 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 1.6 M 2023-01-24T16:36:00.046 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-01-24T16:36:00.046 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 23 k 2023-01-24T16:36:00.046 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 14 k 2023-01-24T16:36:00.047 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-01-24T16:36:00.047 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-24T16:36:00.047 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-24T16:36:00.047 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:00.048 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:36:00.048 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================================== 2023-01-24T16:36:00.048 INFO:teuthology.orchestra.run.smithi097.stdout:Install 6 Packages 2023-01-24T16:36:00.048 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:00.049 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 2.0 M 2023-01-24T16:36:00.049 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 7.8 M 2023-01-24T16:36:00.049 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:36:00.170 INFO:teuthology.orchestra.run.smithi097.stdout:(1/6): ceph-grafana-dashboards-17.2.5-727.g49f8 187 kB/s | 23 kB 00:00 2023-01-24T16:36:00.177 INFO:teuthology.orchestra.run.smithi097.stdout:(2/6): ceph-prometheus-alerts-17.2.5-727.g49f8f 111 kB/s | 14 kB 00:00 2023-01-24T16:36:00.271 INFO:teuthology.orchestra.run.smithi097.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 331 kB/s | 34 kB 00:00 2023-01-24T16:36:00.309 INFO:teuthology.orchestra.run.smithi097.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.1 MB/s | 43 kB 00:00 2023-01-24T16:36:00.368 INFO:teuthology.orchestra.run.smithi097.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2023-01-24T16:36:00.652 INFO:teuthology.orchestra.run.smithi097.stdout:(6/6): ceph-mgr-dashboard-17.2.5-727.g49f8fb05. 2.7 MB/s | 1.6 MB 00:00 2023-01-24T16:36:00.655 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:36:00.655 INFO:teuthology.orchestra.run.smithi097.stdout:Total 3.2 MB/s | 2.0 MB 00:00 2023-01-24T16:36:00.657 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:36:00.664 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:36:00.665 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:36:00.751 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:36:00.755 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:36:01.040 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:36:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-01-24T16:36:01.463 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-01-24T16:36:01.674 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-01-24T16:36:01.802 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-prometheus-alerts-2:17.2.5-727.g49f8fb05.el8.no 4/6 2023-01-24T16:36:02.144 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-grafana-dashboards-2:17.2.5-727.g49f8fb05.el8.n 5/6 2023-01-24T16:36:02.956 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-dashboard-2:17.2.5-727.g49f8fb05.el8.noarch 6/6 2023-01-24T16:36:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.5-727.g49f8fb05.el8.noarch 6/6 2023-01-24T16:36:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-grafana-dashboards-2:17.2.5-727.g49f8fb05.el8.n 1/6 2023-01-24T16:36:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-dashboard-2:17.2.5-727.g49f8fb05.el8.noarch 2/6 2023-01-24T16:36:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-prometheus-alerts-2:17.2.5-727.g49f8fb05.el8.no 3/6 2023-01-24T16:36:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-01-24T16:36:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-01-24T16:36:04.101 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-01-24T16:36:04.101 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:55 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:36:06.171 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:36:06.268 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:06.268 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:36:06.268 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T16:36:06.269 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T16:36:06.269 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T16:36:06.269 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-24T16:36:06.269 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-24T16:36:06.269 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-24T16:36:06.270 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:06.270 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:36:06.384 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-diskprediction-local 2023-01-24T16:36:06.744 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:36:07.668 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:58 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:36:09.433 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:36:09.434 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================================================== 2023-01-24T16:36:09.434 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-01-24T16:36:09.434 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================================================== 2023-01-24T16:36:09.435 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:36:09.435 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 7.4 M 2023-01-24T16:36:09.435 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-01-24T16:36:09.435 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-01-24T16:36:09.436 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-01-24T16:36:09.436 INFO:teuthology.orchestra.run.smithi097.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-01-24T16:36:09.436 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-01-24T16:36:09.436 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-01-24T16:36:09.436 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-01-24T16:36:09.437 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-01-24T16:36:09.437 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:09.437 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:36:09.437 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================================================== 2023-01-24T16:36:09.437 INFO:teuthology.orchestra.run.smithi097.stdout:Install 8 Packages 2023-01-24T16:36:09.438 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:09.439 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 36 M 2023-01-24T16:36:09.440 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 199 M 2023-01-24T16:36:09.440 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:36:09.472 INFO:teuthology.orchestra.run.smithi097.stdout:(1/8): libgfortran-8.4.1-1.el8.x86_64.rpm 18 MB/s | 639 kB 00:00 2023-01-24T16:36:09.473 INFO:teuthology.orchestra.run.smithi097.stdout:(2/8): libquadmath-8.4.1-1.el8.x86_64.rpm 4.4 MB/s | 169 kB 00:00 2023-01-24T16:36:09.492 INFO:teuthology.orchestra.run.smithi097.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 13 MB/s | 225 kB 00:00 2023-01-24T16:36:09.553 INFO:teuthology.orchestra.run.smithi097.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 61 MB/s | 3.7 MB 00:00 2023-01-24T16:36:09.628 INFO:teuthology.orchestra.run.smithi097.stdout:(5/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 63 MB/s | 4.7 MB 00:00 2023-01-24T16:36:09.761 INFO:teuthology.orchestra.run.smithi097.stdout:(6/8): python3-scipy-1.0.0-20.module+el8.1.0+33 50 MB/s | 15 MB 00:00 2023-01-24T16:36:09.800 INFO:teuthology.orchestra.run.smithi097.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 27 MB/s | 4.6 MB 00:00 2023-01-24T16:36:10.144 INFO:teuthology.orchestra.run.smithi097.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.5-727 10 MB/s | 7.4 MB 00:00 2023-01-24T16:36:10.145 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:36:10.146 INFO:teuthology.orchestra.run.smithi097.stdout:Total 51 MB/s | 36 MB 00:00 2023-01-24T16:36:10.155 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:36:10.174 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:36:10.174 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:36:10.634 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:36:10.638 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:36:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:36:11.445 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-01-24T16:36:11.566 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-01-24T16:36:11.753 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-01-24T16:36:12.300 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-01-24T16:36:12.502 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-01-24T16:36:13.040 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-01-24T16:36:13.261 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-01-24T16:36:14.030 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-01-24T16:36:14.362 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-01-24T16:36:14.627 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-01-24T16:36:16.681 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-01-24T16:36:17.952 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-01-24T16:36:18.126 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.5-727.g49f8fb05 8/8 2023-01-24T16:36:18.405 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.5-727.g49f8fb05 8/8 2023-01-24T16:36:18.405 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.5-727.g49f8fb05 1/8 2023-01-24T16:36:18.405 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 2/8 2023-01-24T16:36:18.405 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 3/8 2023-01-24T16:36:18.406 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 4/8 2023-01-24T16:36:18.406 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2023-01-24T16:36:18.406 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2023-01-24T16:36:18.406 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 7/8 2023-01-24T16:36:19.532 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 8/8 2023-01-24T16:36:19.532 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:10 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:36:21.434 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:36:21.530 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:21.530 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:36:21.530 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T16:36:21.530 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-01-24T16:36:21.531 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-01-24T16:36:21.531 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-0.3.12-1.el8.x86_64 2023-01-24T16:36:21.531 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-01-24T16:36:21.531 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-01-24T16:36:21.531 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-01-24T16:36:21.532 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-01-24T16:36:21.532 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:21.532 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:36:21.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-rook 2023-01-24T16:36:22.050 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:36:22.811 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:13 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:36:24.626 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:36:24.627 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================================= 2023-01-24T16:36:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-01-24T16:36:24.628 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================================= 2023-01-24T16:36:24.628 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:36:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 47 k 2023-01-24T16:36:24.629 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-01-24T16:36:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-24T16:36:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-24T16:36:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-24T16:36:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-01-24T16:36:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-01-24T16:36:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-24T16:36:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-01-24T16:36:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-01-24T16:36:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-01-24T16:36:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-01-24T16:36:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-24T16:36:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-24T16:36:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:24.631 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:36:24.632 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================================= 2023-01-24T16:36:24.632 INFO:teuthology.orchestra.run.smithi097.stdout:Install 13 Packages 2023-01-24T16:36:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:24.633 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 1.9 M 2023-01-24T16:36:24.633 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 25 M 2023-01-24T16:36:24.633 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:36:24.715 INFO:teuthology.orchestra.run.smithi097.stdout:(1/13): python3-cachetools-3.1.1-4.el8.noarch.r 391 kB/s | 33 kB 00:00 2023-01-24T16:36:24.716 INFO:teuthology.orchestra.run.smithi097.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 193 kB/s | 16 kB 00:00 2023-01-24T16:36:24.772 INFO:teuthology.orchestra.run.smithi097.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.4 MB/s | 81 kB 00:00 2023-01-24T16:36:24.774 INFO:teuthology.orchestra.run.smithi097.stdout:(4/13): ceph-mgr-rook-17.2.5-727.g49f8fb05.el8. 328 kB/s | 47 kB 00:00 2023-01-24T16:36:24.801 INFO:teuthology.orchestra.run.smithi097.stdout:(5/13): python3-rsa-4.8-1.el8.noarch.rpm 2.0 MB/s | 61 kB 00:00 2023-01-24T16:36:24.831 INFO:teuthology.orchestra.run.smithi097.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 5.2 MB/s | 155 kB 00:00 2023-01-24T16:36:24.846 INFO:teuthology.orchestra.run.smithi097.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 2.8 MB/s | 43 kB 00:00 2023-01-24T16:36:24.862 INFO:teuthology.orchestra.run.smithi097.stdout:(8/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 1.6 MB/s | 27 kB 00:00 2023-01-24T16:36:24.879 INFO:teuthology.orchestra.run.smithi097.stdout:(9/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 7.4 MB/s | 126 kB 00:00 2023-01-24T16:36:24.880 INFO:teuthology.orchestra.run.smithi097.stdout:(10/13): python3-websocket-client-0.56.0-5.el8. 577 kB/s | 61 kB 00:00 2023-01-24T16:36:24.896 INFO:teuthology.orchestra.run.smithi097.stdout:(11/13): python3-pyasn1-modules-0.3.7-6.el8.noa 6.4 MB/s | 110 kB 00:00 2023-01-24T16:36:24.910 INFO:teuthology.orchestra.run.smithi097.stdout:(12/13): python3-jsonpointer-1.10-11.el8.noarch 671 kB/s | 20 kB 00:00 2023-01-24T16:36:24.915 INFO:teuthology.orchestra.run.smithi097.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.9 MB/s | 1.2 MB 00:00 2023-01-24T16:36:24.915 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:36:24.916 INFO:teuthology.orchestra.run.smithi097.stdout:Total 6.7 MB/s | 1.9 MB 00:00 2023-01-24T16:36:24.921 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:36:24.935 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:36:24.935 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:36:25.019 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:36:25.024 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:36:25.539 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:36:25.797 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-01-24T16:36:26.049 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rsa-4.8-1.el8.noarch 2/13 2023-01-24T16:36:26.283 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-01-24T16:36:26.492 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-01-24T16:36:26.774 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-01-24T16:36:27.068 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-01-24T16:36:27.332 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-01-24T16:36:27.568 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-01-24T16:36:27.793 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-01-24T16:36:28.058 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-01-24T16:36:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-01-24T16:36:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-01-24T16:36:29.666 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-rook-2:17.2.5-727.g49f8fb05.el8.noarch 13/13 2023-01-24T16:36:29.957 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.5-727.g49f8fb05.el8.noarch 13/13 2023-01-24T16:36:29.958 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-rook-2:17.2.5-727.g49f8fb05.el8.noarch 1/13 2023-01-24T16:36:29.958 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-01-24T16:36:29.958 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-01-24T16:36:29.958 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-01-24T16:36:29.959 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-01-24T16:36:29.959 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 6/13 2023-01-24T16:36:29.959 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-01-24T16:36:29.959 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-01-24T16:36:29.959 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-01-24T16:36:29.960 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2023-01-24T16:36:29.960 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2023-01-24T16:36:29.960 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/13 2023-01-24T16:36:30.934 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-01-24T16:36:30.934 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:21 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:36:32.835 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:36:32.933 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:32.933 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:36:32.933 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T16:36:32.934 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-24T16:36:32.934 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-24T16:36:32.934 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-24T16:36:32.934 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-24T16:36:32.935 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-24T16:36:32.935 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-24T16:36:32.935 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-24T16:36:32.935 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-24T16:36:32.936 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-24T16:36:32.936 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-24T16:36:32.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-24T16:36:32.937 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-24T16:36:32.937 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:32.937 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:36:33.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-cephadm 2023-01-24T16:36:33.427 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:36:34.152 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:25 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:36:35.912 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:36:35.913 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================== 2023-01-24T16:36:35.913 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-01-24T16:36:35.913 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================== 2023-01-24T16:36:35.914 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:36:35.914 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 115 k 2023-01-24T16:36:35.914 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-01-24T16:36:35.914 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-24T16:36:35.914 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-01-24T16:36:35.915 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-01-24T16:36:35.915 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-24T16:36:35.915 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:35.915 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:36:35.915 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================== 2023-01-24T16:36:35.916 INFO:teuthology.orchestra.run.smithi097.stdout:Install 5 Packages 2023-01-24T16:36:35.916 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:35.916 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 5.9 M 2023-01-24T16:36:35.917 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 26 M 2023-01-24T16:36:35.917 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:36:36.026 INFO:teuthology.orchestra.run.smithi097.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 546 kB/s | 60 kB 00:00 2023-01-24T16:36:36.082 INFO:teuthology.orchestra.run.smithi097.stdout:(2/5): ceph-mgr-cephadm-17.2.5-727.g49f8fb05.el 690 kB/s | 115 kB 00:00 2023-01-24T16:36:36.110 INFO:teuthology.orchestra.run.smithi097.stdout:(3/5): python3-babel-2.5.1-5.el8.noarch.rpm 57 MB/s | 4.8 MB 00:00 2023-01-24T16:36:36.113 INFO:teuthology.orchestra.run.smithi097.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.4 MB/s | 476 kB 00:00 2023-01-24T16:36:36.118 INFO:teuthology.orchestra.run.smithi097.stdout:(5/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 15 MB/s | 538 kB 00:00 2023-01-24T16:36:36.118 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:36:36.118 INFO:teuthology.orchestra.run.smithi097.stdout:Total 29 MB/s | 5.9 MB 00:00 2023-01-24T16:36:36.121 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:36:36.130 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:36:36.130 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:36:36.223 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:36:36.228 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:36:37.214 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:36:37.622 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-01-24T16:36:37.887 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-01-24T16:36:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-01-24T16:36:38.579 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-01-24T16:36:38.774 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 5/5 2023-01-24T16:36:39.035 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 5/5 2023-01-24T16:36:39.036 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/5 2023-01-24T16:36:39.036 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-01-24T16:36:39.036 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-01-24T16:36:39.036 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-01-24T16:36:39.968 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-01-24T16:36:39.969 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:30 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:36:42.284 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:36:42.385 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:42.385 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:36:42.385 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T16:36:42.385 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-24T16:36:42.386 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel-2.5.1-5.el8.noarch 2023-01-24T16:36:42.386 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-01-24T16:36:42.386 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-24T16:36:42.386 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:42.386 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:36:42.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-fuse 2023-01-24T16:36:42.856 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:36:43.748 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:34 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:36:45.515 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:36:45.516 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:36:45.517 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-01-24T16:36:45.517 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:36:45.517 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:36:45.517 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse 2023-01-24T16:36:45.517 INFO:teuthology.orchestra.run.smithi097.stdout: x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 872 k 2023-01-24T16:36:45.518 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-01-24T16:36:45.518 INFO:teuthology.orchestra.run.smithi097.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-01-24T16:36:45.518 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:45.518 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:36:45.518 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:36:45.519 INFO:teuthology.orchestra.run.smithi097.stdout:Install 2 Packages 2023-01-24T16:36:45.519 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:45.519 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 954 k 2023-01-24T16:36:45.519 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 3.0 M 2023-01-24T16:36:45.519 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:36:45.549 INFO:teuthology.orchestra.run.smithi097.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 83 kB 00:00 2023-01-24T16:36:46.230 INFO:teuthology.orchestra.run.smithi097.stdout:(2/2): ceph-fuse-17.2.5-727.g49f8fb05.el8.x86_6 1.2 MB/s | 872 kB 00:00 2023-01-24T16:36:46.231 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:36:46.231 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.3 MB/s | 954 kB 00:00 2023-01-24T16:36:46.233 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:36:46.257 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:36:46.257 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:36:46.346 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:36:46.350 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:36:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:36:46.790 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-01-24T16:36:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 2/2 2023-01-24T16:36:47.362 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 2/2 2023-01-24T16:36:47.363 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 1/2 2023-01-24T16:36:48.293 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-01-24T16:36:48.293 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:39 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:36:50.424 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:36:50.521 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:50.521 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:36:50.521 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-01-24T16:36:50.521 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:50.522 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:36:50.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-volume 2023-01-24T16:36:50.992 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:36:51.807 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:42 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:36:53.587 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:36:53.588 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:36:53.589 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-01-24T16:36:53.589 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:36:53.590 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:36:53.590 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 263 k 2023-01-24T16:36:53.590 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:53.591 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:36:53.591 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:36:53.591 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-01-24T16:36:53.591 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:53.592 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 263 k 2023-01-24T16:36:53.592 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 1.3 M 2023-01-24T16:36:53.593 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:36:53.798 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-volume-17.2.5-727.g49f8fb05.el8.noarch.rpm 1.2 MB/s | 263 kB 00:00 2023-01-24T16:36:53.799 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:36:53.799 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.2 MB/s | 263 kB 00:00 2023-01-24T16:36:53.799 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:36:53.809 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:36:53.810 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:36:53.832 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:36:53.836 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:36:54.020 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:36:54.356 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-volume-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T16:36:54.732 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-volume-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T16:36:55.650 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-volume-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T16:36:55.650 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:46 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:36:57.661 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:36:57.757 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:57.757 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:36:57.757 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T16:36:57.757 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:36:57.758 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:36:57.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados-devel 2023-01-24T16:36:58.271 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:36:59.123 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:50 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:00.895 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:37:00.896 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:00.896 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-01-24T16:37:00.896 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:00.897 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:37:00.897 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 130 k 2023-01-24T16:37:00.897 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:00.897 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:37:00.897 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:00.898 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-01-24T16:37:00.898 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:00.898 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 130 k 2023-01-24T16:37:00.898 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 464 k 2023-01-24T16:37:00.899 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:37:01.414 INFO:teuthology.orchestra.run.smithi097.stdout:librados-devel-17.2.5-727.g49f8fb05.el8.x86_64. 251 kB/s | 130 kB 00:00 2023-01-24T16:37:01.415 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:37:01.415 INFO:teuthology.orchestra.run.smithi097.stdout:Total 250 kB/s | 130 kB 00:00 2023-01-24T16:37:01.416 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:37:01.423 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:37:01.423 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:37:01.481 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:37:01.485 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:37:01.710 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:37:01.995 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:02.307 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:03.188 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:54 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:05.361 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:37:05.456 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:05.456 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:37:05.456 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:37:05.457 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:05.457 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:37:05.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs2 2023-01-24T16:37:05.961 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:37:06.752 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:57 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:08.107 INFO:teuthology.orchestra.run.smithi097.stdout:Package libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T16:37:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:37:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-01-24T16:37:08.557 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:37:08.656 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs-devel 2023-01-24T16:37:09.019 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:37:09.820 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:00 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:11.617 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:37:11.618 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:11.618 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-01-24T16:37:11.619 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:11.619 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:37:11.619 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 25 k 2023-01-24T16:37:11.619 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:11.619 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:37:11.619 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:11.620 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-01-24T16:37:11.620 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:11.620 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 25 k 2023-01-24T16:37:11.620 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 109 k 2023-01-24T16:37:11.620 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:37:11.740 INFO:teuthology.orchestra.run.smithi097.stdout:libcephfs-devel-17.2.5-727.g49f8fb05.el8.x86_64 204 kB/s | 25 kB 00:00 2023-01-24T16:37:11.741 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:37:11.742 INFO:teuthology.orchestra.run.smithi097.stdout:Total 201 kB/s | 25 kB 00:00 2023-01-24T16:37:11.742 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:37:11.745 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:37:11.745 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:37:11.747 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:37:11.751 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:37:11.808 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:37:11.993 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:12.235 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:13.139 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:13.139 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:04 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:15.355 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:37:15.453 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:15.453 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:37:15.454 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:37:15.454 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:15.454 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:37:15.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados2 2023-01-24T16:37:15.953 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:37:16.713 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:07 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:18.027 INFO:teuthology.orchestra.run.smithi097.stdout:Package librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T16:37:18.475 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:37:18.477 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-01-24T16:37:18.477 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:37:18.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librbd1 2023-01-24T16:37:18.937 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:37:19.837 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:10 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:21.185 INFO:teuthology.orchestra.run.smithi097.stdout:Package librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T16:37:21.642 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:37:21.643 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-01-24T16:37:21.644 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:37:21.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rados 2023-01-24T16:37:22.099 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:37:23.066 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:14 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:24.396 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rados-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T16:37:24.841 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:37:24.842 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-01-24T16:37:24.842 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:37:24.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rgw 2023-01-24T16:37:25.299 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:37:26.106 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:17 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:27.401 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rgw-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T16:37:27.856 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:37:27.857 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-01-24T16:37:27.857 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:37:27.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-cephfs 2023-01-24T16:37:28.322 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:37:29.120 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:20 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:30.442 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-cephfs-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T16:37:30.901 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:37:30.902 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-01-24T16:37:30.902 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:37:31.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rbd 2023-01-24T16:37:31.367 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:37:32.154 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:23 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:33.526 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rbd-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T16:37:33.972 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:37:33.973 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-01-24T16:37:33.973 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:37:34.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-fuse 2023-01-24T16:37:34.423 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:37:35.247 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:26 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:37.057 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:37:37.058 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:37.059 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2023-01-24T16:37:37.059 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:37.059 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:37:37.059 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 90 k 2023-01-24T16:37:37.060 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:37.060 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:37:37.061 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:37.061 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-01-24T16:37:37.061 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:37.061 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 90 k 2023-01-24T16:37:37.062 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 264 k 2023-01-24T16:37:37.062 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:37:37.204 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-fuse-17.2.5-727.g49f8fb05.el8.x86_64.rpm 623 kB/s | 90 kB 00:00 2023-01-24T16:37:37.205 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:37:37.205 INFO:teuthology.orchestra.run.smithi097.stdout:Total 617 kB/s | 90 kB 00:00 2023-01-24T16:37:37.205 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:37:37.216 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:37:37.217 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:37:37.271 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:37:37.275 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:37:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:37:37.599 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:37.848 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:38.724 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:38.725 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:29 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:40.743 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:37:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:40.841 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:37:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:37:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:40.841 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:37:40.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-mirror 2023-01-24T16:37:41.312 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:37:42.089 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:33 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:43.848 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:37:43.850 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:43.850 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-01-24T16:37:43.850 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:43.851 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:37:43.851 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 3.5 M 2023-01-24T16:37:43.851 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:43.852 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:37:43.852 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:43.852 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-01-24T16:37:43.853 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:43.853 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 3.5 M 2023-01-24T16:37:43.853 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 16 M 2023-01-24T16:37:43.853 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:37:44.269 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-mirror-17.2.5-727.g49f8fb05.el8.x86_64.rpm 8.3 MB/s | 3.5 MB 00:00 2023-01-24T16:37:44.270 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:37:44.271 INFO:teuthology.orchestra.run.smithi097.stdout:Total 8.2 MB/s | 3.5 MB 00:00 2023-01-24T16:37:44.271 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:37:44.285 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:37:44.285 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:37:44.353 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:37:44.357 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:37:44.802 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:37:45.024 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:45.430 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:46.386 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:46.387 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:37 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:48.432 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:37:48.528 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:48.528 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:37:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:37:48.529 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:48.529 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:37:48.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-nbd 2023-01-24T16:37:49.048 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:37:49.787 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:40 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:51.621 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:37:51.622 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:51.622 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2023-01-24T16:37:51.622 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:51.623 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:37:51.623 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 172 k 2023-01-24T16:37:51.623 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:51.623 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:37:51.623 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:37:51.624 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-01-24T16:37:51.624 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:51.624 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 172 k 2023-01-24T16:37:51.624 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 548 k 2023-01-24T16:37:51.624 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:37:51.790 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-nbd-17.2.5-727.g49f8fb05.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2023-01-24T16:37:51.791 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:37:51.791 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.0 MB/s | 172 kB 00:00 2023-01-24T16:37:51.792 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:37:51.803 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:37:51.804 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:37:51.858 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:37:51.862 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:37:52.002 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:37:52.172 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:52.446 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-nbd-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:53.340 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T16:37:53.340 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:44 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:37:55.629 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:37:55.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:55.726 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:37:55.727 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T16:37:55.727 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:55.727 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:37:55.846 DEBUG:teuthology.parallel:result is None 2023-01-24T16:37:55.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo lsb_release -is 2023-01-24T16:37:55.884 INFO:teuthology.orchestra.run.smithi097.stdout:RedHatEnterprise 2023-01-24T16:37:55.885 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2023-01-24T16:37:55.886 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T16:37:56.033 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-24T16:37:56.053 INFO:teuthology.orchestra.run.smithi097.stdout:17.2.5-727.g49f8fb05.el8 2023-01-24T16:37:56.054 INFO:teuthology.packaging:The installed version of ceph is 17.2.5-727.g49f8fb05.el8 2023-01-24T16:37:56.054 INFO:teuthology.task.install:The correct ceph version 17.2.5-727.g49f8fb05 is installed. 2023-01-24T16:37:56.055 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-24T16:37:56.055 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:37:56.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-24T16:37:56.117 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-24T16:37:56.118 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:37:56.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2023-01-24T16:37:56.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-24T16:37:56.245 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-24T16:37:56.245 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:37:56.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-24T16:37:56.308 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-24T16:37:56.372 INFO:teuthology.run_tasks:Running task ceph... 2023-01-24T16:37:56.452 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-01-24T16:37:56.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /var/log/ceph 2023-01-24T16:37:56.476 INFO:tasks.ceph:Disabling ceph logrotate... 2023-01-24T16:37:56.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-24T16:37:56.540 INFO:tasks.ceph:Creating extra log directories... 2023-01-24T16:37:56.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-24T16:37:56.611 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-01-24T16:37:56.611 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'setgroup': 'ceph', 'setuser': 'ceph', 'storage classes': 'LUKEWARM, FROZEN'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-01-24T16:37:56.612 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135445', 'branch': 'wip-yuri7-testing-2023-01-23-1532-quincy', 'description': 'rgw/lifecycle/{cluster overrides supported-random-distro$/{rhel_8} tasks/rgw_s3tests}', 'email': 'yweinstein@radhat.com', 'first_in_suite': False, 'job_id': '7135445', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 1, 'size': 10}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri7-testing-2023-01-23-1532-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'setgroup': 'ceph', 'setuser': 'ceph', 'storage classes': 'LUKEWARM, FROZEN'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}}, 'rgw': {'frontend': 'beast', 'storage classes': 'LUKEWARM, FROZEN'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri7-testing-2023-01-23-1532-quincy', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'osd.2', 'mgr.0', 'client.0']], 'sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuri7-testing-2023-01-23-1532-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'targets': {'smithi097.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDH2P9SGj7EEEyNLxYTnBNzzYgzRuF3LWwwm4A9kPUSZVETbWpQyGzTXkCuC8JGJK6hlTXwbQxL5RzTNGiyXbtWfZ8owkv2lN9OCXtsbG07syuCg0IfNd6/dx67qXikCQYA/mW/2165tx0Lr1tzPApYcOIB6SqduX3cfs5jzos3o+HU6VP9UnmjBd3O6w2aw4RSdaHknpq9CjT9Qm5HuCr1hNlWu9p7OYOXkIYLK9LmNfVQSAA3Rz50THlCBRIlNQg1Boh1RfXChWiBFJ88jEKBm96E2R9oYQXb6FQmBujO5IzktunT/CSgheILfDyOX/Xj/h5j0ZOdJICPq+kr83mzM4dqPzV/b2bJbOWlntqi7JeJz7vZuHWHG3MWiek9RPFxn2FWO4Cv0VpDJ/Ttvq5F8C3TP1dEiMhUB7xpwk1eWSsb8tKAX1cyvHZXC8RPL/TM+qAlTYBR1qrTwNRVZLJf+ygsQ2wSeVw2ZCHbTmoL1q3ML2W+HHhVHifXVSMV5es='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'rgw': ['client.0']}, {'s3tests': {'client.0': {'extra_attrs': ['lifecycle'], 'force-branch': 'ceph-quincy', 'lc_debug_interval': 10, 'rgw_server': 'client.0', 'storage classes': 'LUKEWARM, FROZEN'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8e1bba12e584b7ae912e92e94afe9c075ab51fa3', 'timestamp': '2023-01-24_16:04:03', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.17735'} 2023-01-24T16:37:56.612 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-24T16:37:56.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/run/ceph 2023-01-24T16:37:56.720 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:37:56.721 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2023-01-24T16:37:56.775 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-24T16:37:56.776 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2023-01-24T16:37:56.830 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-24T16:37:56.830 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T16:37:56.830 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 6h/6d Inode: 198461 Links: 1 2023-01-24T16:37:56.831 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T16:37:56.831 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T16:37:56.831 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-01-24 16:37:45.202480819 +0000 2023-01-24T16:37:56.831 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-01-24 16:33:26.326670337 +0000 2023-01-24T16:37:56.831 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-01-24 16:33:26.326670337 +0000 2023-01-24T16:37:56.832 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-01-24T16:37:56.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-24T16:37:56.892 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-01-24T16:37:56.893 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-01-24T16:37:56.893 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000229096 s, 2.2 MB/s 2023-01-24T16:37:56.894 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-24T16:37:56.949 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2023-01-24T16:37:57.003 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-24T16:37:57.004 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T16:37:57.004 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 6h/6d Inode: 205909 Links: 1 2023-01-24T16:37:57.004 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T16:37:57.005 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T16:37:57.005 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-01-24 16:37:45.202480819 +0000 2023-01-24T16:37:57.005 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-01-24 16:33:26.837661312 +0000 2023-01-24T16:37:57.005 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-01-24 16:33:26.837661312 +0000 2023-01-24T16:37:57.005 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-01-24T16:37:57.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-24T16:37:57.065 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-01-24T16:37:57.065 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-01-24T16:37:57.066 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000220675 s, 2.3 MB/s 2023-01-24T16:37:57.066 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-24T16:37:57.121 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2023-01-24T16:37:57.175 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-24T16:37:57.175 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T16:37:57.176 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 6h/6d Inode: 206076 Links: 1 2023-01-24T16:37:57.176 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T16:37:57.176 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T16:37:57.176 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-01-24 16:37:45.203480802 +0000 2023-01-24T16:37:57.177 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-01-24 16:33:27.333652551 +0000 2023-01-24T16:37:57.177 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-01-24 16:33:27.333652551 +0000 2023-01-24T16:37:57.177 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-01-24T16:37:57.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-24T16:37:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-01-24T16:37:57.238 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-01-24T16:37:57.238 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000198308 s, 2.6 MB/s 2023-01-24T16:37:57.239 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-24T16:37:57.294 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2023-01-24T16:37:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-24T16:37:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T16:37:57.349 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 6h/6d Inode: 206942 Links: 1 2023-01-24T16:37:57.349 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T16:37:57.350 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T16:37:57.350 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-01-24 16:37:45.203480802 +0000 2023-01-24T16:37:57.350 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-01-24 16:33:27.983641072 +0000 2023-01-24T16:37:57.350 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-01-24 16:33:27.983641072 +0000 2023-01-24T16:37:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-01-24T16:37:57.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-24T16:37:57.411 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-01-24T16:37:57.411 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-01-24T16:37:57.411 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000191981 s, 2.7 MB/s 2023-01-24T16:37:57.412 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-24T16:37:57.467 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T16:37:57.467 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-01-24T16:37:57.468 INFO:tasks.ceph:Generating config... 2023-01-24T16:37:57.469 INFO:tasks.ceph:[client] debug rgw = 20 2023-01-24T16:37:57.469 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-01-24T16:37:57.470 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-01-24T16:37:57.470 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-01-24T16:37:57.470 INFO:tasks.ceph:[client] rgw lc debug interval = 10 2023-01-24T16:37:57.470 INFO:tasks.ceph:[client] setgroup = ceph 2023-01-24T16:37:57.471 INFO:tasks.ceph:[client] setuser = ceph 2023-01-24T16:37:57.471 INFO:tasks.ceph:[client] storage classes = LUKEWARM, FROZEN 2023-01-24T16:37:57.471 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-01-24T16:37:57.471 INFO:tasks.ceph:[mgr] debug ms = 1 2023-01-24T16:37:57.472 INFO:tasks.ceph:[mon] debug mon = 20 2023-01-24T16:37:57.472 INFO:tasks.ceph:[mon] debug ms = 1 2023-01-24T16:37:57.472 INFO:tasks.ceph:[mon] debug paxos = 20 2023-01-24T16:37:57.473 INFO:tasks.ceph:[osd] debug ms = 1 2023-01-24T16:37:57.473 INFO:tasks.ceph:[osd] debug osd = 20 2023-01-24T16:37:57.473 INFO:tasks.ceph:Setting up mon.a... 2023-01-24T16:37:57.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-01-24T16:37:57.553 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.keyring 2023-01-24T16:37:57.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-01-24T16:37:57.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T16:37:57.657 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.97')] 2023-01-24T16:37:57.657 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.97'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'setgroup': 'ceph', 'setuser': 'ceph', 'storage classes': 'LUKEWARM, FROZEN'}, 'mon.a': {}} 2023-01-24T16:37:57.659 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:37:57.659 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T16:37:57.711 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.97 --print /home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:37:57.785 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:37:57.786 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:37:57.787 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:37:57.787 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: generated fsid ba9bee0e-b74d-4668-a675-133390900940 2023-01-24T16:37:57.787 INFO:teuthology.orchestra.run.smithi097.stdout:setting min_mon_release = octopus 2023-01-24T16:37:57.787 INFO:teuthology.orchestra.run.smithi097.stdout:epoch 0 2023-01-24T16:37:57.787 INFO:teuthology.orchestra.run.smithi097.stdout:fsid ba9bee0e-b74d-4668-a675-133390900940 2023-01-24T16:37:57.788 INFO:teuthology.orchestra.run.smithi097.stdout:last_changed 2023-01-24T16:37:57.786171+0000 2023-01-24T16:37:57.788 INFO:teuthology.orchestra.run.smithi097.stdout:created 2023-01-24T16:37:57.786171+0000 2023-01-24T16:37:57.788 INFO:teuthology.orchestra.run.smithi097.stdout:min_mon_release 15 (octopus) 2023-01-24T16:37:57.788 INFO:teuthology.orchestra.run.smithi097.stdout:election_strategy: 1 2023-01-24T16:37:57.789 INFO:teuthology.orchestra.run.smithi097.stdout:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-01-24T16:37:57.789 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-01-24T16:37:57.789 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T16:37:57.841 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ba9bee0e-b74d-4668-a675-133390900940... 2023-01-24T16:37:57.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-24T16:37:57.922 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2023-01-24T16:37:57.922 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2023-01-24T16:37:57.922 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-24T16:37:57.923 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2023-01-24T16:37:57.923 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.923 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2023-01-24T16:37:57.923 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.923 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2023-01-24T16:37:57.924 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.924 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2023-01-24T16:37:57.924 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2023-01-24T16:37:57.924 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.924 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2023-01-24T16:37:57.925 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2023-01-24T16:37:57.925 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.925 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-24T16:37:57.925 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2023-01-24T16:37:57.925 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.926 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2023-01-24T16:37:57.926 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-24T16:37:57.926 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.926 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2023-01-24T16:37:57.927 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.927 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2023-01-24T16:37:57.927 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2023-01-24T16:37:57.927 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2023-01-24T16:37:57.927 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2023-01-24T16:37:57.928 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2023-01-24T16:37:57.928 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2023-01-24T16:37:57.928 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2023-01-24T16:37:57.928 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-24T16:37:57.928 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-24T16:37:57.929 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2023-01-24T16:37:57.929 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.929 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-24T16:37:57.929 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.929 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2023-01-24T16:37:57.930 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.930 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2023-01-24T16:37:57.930 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.930 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2023-01-24T16:37:57.930 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2023-01-24T16:37:57.930 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2023-01-24T16:37:57.931 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = 172.21.15.97 2023-01-24T16:37:57.931 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = ba9bee0e-b74d-4668-a675-133390900940 2023-01-24T16:37:57.931 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.931 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2023-01-24T16:37:57.931 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2023-01-24T16:37:57.932 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.932 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2023-01-24T16:37:57.932 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 600 2023-01-24T16:37:57.932 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.932 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2023-01-24T16:37:57.933 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2023-01-24T16:37:57.933 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.933 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2023-01-24T16:37:57.933 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2023-01-24T16:37:57.933 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2023-01-24T16:37:57.934 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.934 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2023-01-24T16:37:57.934 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2023-01-24T16:37:57.934 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.934 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2023-01-24T16:37:57.934 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.935 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2023-01-24T16:37:57.935 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.935 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2023-01-24T16:37:57.935 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.935 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2023-01-24T16:37:57.936 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2023-01-24T16:37:57.936 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.936 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2023-01-24T16:37:57.936 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2023-01-24T16:37:57.936 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2023-01-24T16:37:57.937 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2023-01-24T16:37:57.937 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.937 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2023-01-24T16:37:57.937 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2023-01-24T16:37:57.937 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2023-01-24T16:37:57.938 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2023-01-24T16:37:57.938 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2023-01-24T16:37:57.938 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2023-01-24T16:37:57.938 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2023-01-24T16:37:57.938 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2023-01-24T16:37:57.938 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.939 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2023-01-24T16:37:57.939 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2023-01-24T16:37:57.939 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2023-01-24T16:37:57.939 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2023-01-24T16:37:57.939 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2023-01-24T16:37:57.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2023-01-24T16:37:57.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2023-01-24T16:37:57.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2023-01-24T16:37:57.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2023-01-24T16:37:57.940 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2023-01-24T16:37:57.941 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2023-01-24T16:37:57.941 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.941 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-24T16:37:57.941 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2023-01-24T16:37:57.941 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2023-01-24T16:37:57.941 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.942 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2023-01-24T16:37:57.942 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-24T16:37:57.942 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-24T16:37:57.942 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.942 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2023-01-24T16:37:57.943 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2023-01-24T16:37:57.943 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.943 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2023-01-24T16:37:57.943 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:37:57.943 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2023-01-24T16:37:57.943 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2023-01-24T16:37:57.944 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2023-01-24T16:37:57.944 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2023-01-24T16:37:57.944 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-24T16:37:57.944 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-24T16:37:57.944 INFO:teuthology.orchestra.run.smithi097.stdout: debug rgw = 20 2023-01-24T16:37:57.945 INFO:teuthology.orchestra.run.smithi097.stdout: rgw crypt require ssl = False 2023-01-24T16:37:57.945 INFO:teuthology.orchestra.run.smithi097.stdout: rgw crypt s3 kms backend = testing 2023-01-24T16:37:57.945 INFO:teuthology.orchestra.run.smithi097.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-01-24T16:37:57.945 INFO:teuthology.orchestra.run.smithi097.stdout: rgw lc debug interval = 10 2023-01-24T16:37:57.945 INFO:teuthology.orchestra.run.smithi097.stdout: setgroup = ceph 2023-01-24T16:37:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: setuser = ceph 2023-01-24T16:37:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: storage classes = "LUKEWARM, FROZEN" 2023-01-24T16:37:57.946 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2023-01-24T16:37:57.947 INFO:tasks.ceph:Creating admin key on mon.a... 2023-01-24T16:37:57.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-01-24T16:37:58.037 INFO:tasks.ceph:Copying monmap to all nodes... 2023-01-24T16:37:58.037 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:37:58.038 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-01-24T16:37:58.053 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:37:58.053 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-01-24T16:37:58.106 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi097.front.sepia.ceph.com 2023-01-24T16:37:58.107 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:37:58.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-24T16:37:58.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T16:37:58.179 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:37:58.179 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:37:58.233 INFO:tasks.ceph:Setting up mon nodes... 2023-01-24T16:37:58.234 INFO:tasks.ceph:Setting up mgr nodes... 2023-01-24T16:37:58.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mgr/ceph-0 && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.0 /var/lib/ceph/mgr/ceph-0/keyring 2023-01-24T16:37:58.319 INFO:teuthology.orchestra.run.smithi097.stdout:creating /var/lib/ceph/mgr/ceph-0/keyring 2023-01-24T16:37:58.321 INFO:tasks.ceph:Setting up mds nodes... 2023-01-24T16:37:58.321 INFO:tasks.ceph_client:Setting up client nodes... 2023-01-24T16:37:58.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-01-24T16:37:58.359 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-01-24T16:37:58.371 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-01-24T16:37:58.371 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-01-24T16:37:58.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-01-24T16:37:58.435 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T16:37:58.435 INFO:tasks.ceph:role: osd.0 2023-01-24T16:37:58.435 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com 2023-01-24T16:37:58.436 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-24T16:37:58.506 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T16:37:58.507 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T16:37:58.507 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T16:37:58.508 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 2023-01-24T16:37:58.508 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T16:37:58.509 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2023-01-24T16:37:58.509 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T16:37:58.510 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T16:37:58.510 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T16:37:58.510 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T16:37:58.873 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2023-01-24T16:37:58.874 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2023-01-24T16:37:58.875 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-01-24T16:37:59.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-01-24T16:37:59.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-01-24T16:37:59.108 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T16:37:59.109 INFO:tasks.ceph:role: osd.1 2023-01-24T16:37:59.109 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com 2023-01-24T16:37:59.110 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-24T16:37:59.178 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T16:37:59.179 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T16:37:59.179 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T16:37:59.179 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 2023-01-24T16:37:59.179 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T16:37:59.179 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2023-01-24T16:37:59.180 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T16:37:59.180 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T16:37:59.180 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T16:37:59.180 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T16:37:59.537 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2023-01-24T16:37:59.538 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2023-01-24T16:37:59.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-01-24T16:37:59.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-01-24T16:37:59.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-01-24T16:37:59.715 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T16:37:59.716 INFO:tasks.ceph:role: osd.2 2023-01-24T16:37:59.716 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi097.front.sepia.ceph.com 2023-01-24T16:37:59.716 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-01-24T16:37:59.786 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T16:37:59.786 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T16:37:59.786 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T16:37:59.787 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 2023-01-24T16:37:59.787 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T16:37:59.787 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2023-01-24T16:37:59.787 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T16:37:59.787 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T16:37:59.788 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T16:37:59.788 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T16:38:00.150 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2023-01-24T16:38:00.151 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2023-01-24T16:38:00.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-01-24T16:38:00.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-01-24T16:38:00.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:38:00.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:00.362+0000 7f0b36fda540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-01-24T16:38:00.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:00.365+0000 7f0b36fda540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-01-24T16:38:00.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:00.365+0000 7f0b36fda540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T16:38:00.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:00.365+0000 7f0b36fda540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T16:38:00.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:00.365+0000 7f0b36fda540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T16:38:00.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:00.366+0000 7f0b36fda540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-01-24T16:38:02.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-01-24T16:38:02.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:38:02.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:02.598+0000 7fe2a2c2d540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-01-24T16:38:02.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:02.598+0000 7fe2a2c2d540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-01-24T16:38:02.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:02.598+0000 7fe2a2c2d540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T16:38:02.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:02.598+0000 7fe2a2c2d540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T16:38:02.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:02.598+0000 7fe2a2c2d540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T16:38:02.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:02.598+0000 7fe2a2c2d540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-01-24T16:38:04.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-01-24T16:38:04.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:38:04.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:04.793+0000 7f112b1d1540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-01-24T16:38:04.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:04.793+0000 7f112b1d1540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-01-24T16:38:04.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:04.793+0000 7f112b1d1540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T16:38:04.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:04.793+0000 7f112b1d1540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T16:38:04.795 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:04.793+0000 7f112b1d1540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T16:38:04.795 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:04.793+0000 7f112b1d1540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-01-24T16:38:06.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-01-24T16:38:06.899 INFO:tasks.ceph:Reading keys from all nodes... 2023-01-24T16:38:06.900 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:38:06.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/mgr/ceph-0/keyring of=/dev/stdout 2023-01-24T16:38:06.965 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:38:06.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-01-24T16:38:07.030 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:38:07.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-01-24T16:38:07.094 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:38:07.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-01-24T16:38:07.158 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:38:07.158 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-01-24T16:38:07.212 INFO:tasks.ceph:Adding keys to all mons... 2023-01-24T16:38:07.213 DEBUG:teuthology.orchestra.run.smithi097:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-24T16:38:07.274 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr.0] 2023-01-24T16:38:07.275 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBmCdBjgj0NExAAaPk76bF/dij+DUvYwNOU2A== 2023-01-24T16:38:07.275 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.0] 2023-01-24T16:38:07.275 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBoCdBjkKamFRAAh+lvtg/2aHcpt82F2fH2Ag== 2023-01-24T16:38:07.276 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.1] 2023-01-24T16:38:07.276 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBqCdBjP+6xIxAAb5BhMezCKKq6uy3VeK5aBQ== 2023-01-24T16:38:07.276 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.2] 2023-01-24T16:38:07.276 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBsCdBjvElSLxAAgjOKJOok09YCtAH4tkGYXg== 2023-01-24T16:38:07.276 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2023-01-24T16:38:07.277 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBmCdBj+mBxFRAAxArnaFtvfOJoFFgNN6vCIw== 2023-01-24T16:38:07.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.0 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-24T16:38:07.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T16:38:07.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T16:38:07.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T16:38:07.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-24T16:38:07.549 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-01-24T16:38:07.549 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-01-24T16:38:07.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-24T16:38:07.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-01-24T16:38:07.685 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:38:07.739 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-01-24T16:38:07.740 INFO:tasks.ceph.mon.a:Restarting daemon 2023-01-24T16:38:07.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-01-24T16:38:07.781 INFO:tasks.ceph.mon.a:Started 2023-01-24T16:38:07.782 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-01-24T16:38:07.782 INFO:tasks.ceph.mgr.0:Restarting daemon 2023-01-24T16:38:07.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i 0 2023-01-24T16:38:07.785 INFO:tasks.ceph.mgr.0:Started 2023-01-24T16:38:07.785 INFO:tasks.ceph:Setting crush tunables to default 2023-01-24T16:38:07.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd crush tunables default 2023-01-24T16:38:08.001 INFO:teuthology.orchestra.run.smithi097.stderr:adjusted tunables profile to default 2023-01-24T16:38:08.012 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-01-24T16:38:08.012 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:38:08.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-01-24T16:38:08.035 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:38:08.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-01-24T16:38:08.099 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:38:08.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-01-24T16:38:08.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd new a9a7d912-dc32-4e2c-a692-97890541722c 0 2023-01-24T16:38:08.410 INFO:teuthology.orchestra.run.smithi097.stdout:0 2023-01-24T16:38:08.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd new 3719c1fb-f16b-4fa9-8110-43052f05c318 1 2023-01-24T16:38:08.628 INFO:teuthology.orchestra.run.smithi097.stdout:1 2023-01-24T16:38:08.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd new 3c859fd8-83e0-478d-8766-276c16357101 2 2023-01-24T16:38:08.849 INFO:teuthology.orchestra.run.smithi097.stdout:2 2023-01-24T16:38:08.861 INFO:tasks.ceph.osd.0:Restarting daemon 2023-01-24T16:38:08.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-01-24T16:38:08.863 INFO:tasks.ceph.osd.0:Started 2023-01-24T16:38:08.864 INFO:tasks.ceph.osd.1:Restarting daemon 2023-01-24T16:38:08.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-01-24T16:38:08.866 INFO:tasks.ceph.osd.1:Started 2023-01-24T16:38:08.867 INFO:tasks.ceph.osd.2:Restarting daemon 2023-01-24T16:38:08.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-01-24T16:38:08.869 INFO:tasks.ceph.osd.2:Started 2023-01-24T16:38:08.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T16:38:09.084 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:09.085 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":5,"fsid":"ba9bee0e-b74d-4668-a675-133390900940","created":"2023-01-24T16:38:07.892199+0000","modified":"2023-01-24T16:38:08.842585+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T16:38:08.842585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a9a7d912-dc32-4e2c-a692-97890541722c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3719c1fb-f16b-4fa9-8110-43052f05c318","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3c859fd8-83e0-478d-8766-276c16357101","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T16:38:09.096 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-01-24T16:38:09.097 INFO:tasks.ceph:Waiting for OSDs to come up 2023-01-24T16:38:09.223 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:09.223+0000 7ff572c0f180 -1 mgr[py] Module not found: 'prometheus' 2023-01-24T16:38:09.224 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:09.223+0000 7ff572c0f180 -1 mgr[py] Traceback (most recent call last): 2023-01-24T16:38:09.224 INFO:tasks.ceph.mgr.0.smithi097.stderr: File "/usr/share/ceph/mgr/prometheus/__init__.py", line 2, in 2023-01-24T16:38:09.225 INFO:tasks.ceph.mgr.0.smithi097.stderr: from .module import Module, StandbyModule 2023-01-24T16:38:09.225 INFO:tasks.ceph.mgr.0.smithi097.stderr: File "/usr/share/ceph/mgr/prometheus/module.py", line 3, in 2023-01-24T16:38:09.225 INFO:tasks.ceph.mgr.0.smithi097.stderr: from packaging.version import Version 2023-01-24T16:38:09.225 INFO:tasks.ceph.mgr.0.smithi097.stderr:ModuleNotFoundError: No module named 'packaging' 2023-01-24T16:38:09.225 INFO:tasks.ceph.mgr.0.smithi097.stderr: 2023-01-24T16:38:09.226 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:09.223+0000 7ff572c0f180 -1 mgr[py] Class not found in module 'prometheus' 2023-01-24T16:38:09.226 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:09.223+0000 7ff572c0f180 -1 mgr[py] Error loading module 'prometheus': (2) No such file or directory 2023-01-24T16:38:09.352 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:09.352+0000 7ff572c0f180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-24T16:38:09.399 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T16:38:09.426 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:09.426+0000 7ff572c0f180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-24T16:38:09.460 INFO:tasks.ceph.osd.1.smithi097.stderr:2023-01-24T16:38:09.459+0000 7fc0c0dd2540 -1 Falling back to public interface 2023-01-24T16:38:09.467 INFO:tasks.ceph.osd.0.smithi097.stderr:2023-01-24T16:38:09.466+0000 7faa86aad540 -1 Falling back to public interface 2023-01-24T16:38:09.483 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-01-24T16:38:09.482+0000 7f5049c2b540 -1 Falling back to public interface 2023-01-24T16:38:09.582 INFO:teuthology.misc.health.smithi097.stdout: 2023-01-24T16:38:09.583 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":5,"fsid":"ba9bee0e-b74d-4668-a675-133390900940","created":"2023-01-24T16:38:07.892199+0000","modified":"2023-01-24T16:38:08.842585+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T16:38:08.842585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a9a7d912-dc32-4e2c-a692-97890541722c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3719c1fb-f16b-4fa9-8110-43052f05c318","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3c859fd8-83e0-478d-8766-276c16357101","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T16:38:09.593 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-01-24T16:38:09.617 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:09.617+0000 7ff572c0f180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-24T16:38:09.786 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:09.786+0000 7ff572c0f180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-24T16:38:09.861 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:09.861+0000 7ff572c0f180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-24T16:38:09.993 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:09.992+0000 7ff572c0f180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-24T16:38:10.087 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:10.086+0000 7ff572c0f180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-24T16:38:10.582 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:10.582+0000 7ff572c0f180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-24T16:38:10.658 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:10.658+0000 7ff572c0f180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-24T16:38:10.734 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:10.734+0000 7ff572c0f180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-24T16:38:11.298 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:11.297+0000 7ff572c0f180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-01-24T16:38:11.433 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:11.433+0000 7ff572c0f180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-24T16:38:11.530 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:11.529+0000 7ff572c0f180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-24T16:38:11.772 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:11.771+0000 7ff572c0f180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-24T16:38:11.846 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:11.845+0000 7ff572c0f180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-24T16:38:12.225 INFO:tasks.ceph.osd.1.smithi097.stderr:2023-01-24T16:38:12.225+0000 7fc0c0dd2540 -1 osd.1 0 log_to_monitors true 2023-01-24T16:38:12.228 INFO:tasks.ceph.osd.0.smithi097.stderr:2023-01-24T16:38:12.228+0000 7faa86aad540 -1 osd.0 0 log_to_monitors true 2023-01-24T16:38:12.246 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-01-24T16:38:12.246+0000 7f5049c2b540 -1 osd.2 0 log_to_monitors true 2023-01-24T16:38:12.425 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:12.424+0000 7ff572c0f180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-24T16:38:12.682 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:12.682+0000 7ff572c0f180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-24T16:38:12.813 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:12.812+0000 7ff572c0f180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-24T16:38:12.896 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:12.895+0000 7ff572c0f180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-24T16:38:13.041 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:13.041+0000 7ff572c0f180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-24T16:38:13.114 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:13.114+0000 7ff572c0f180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-24T16:38:13.187 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:13.187+0000 7ff572c0f180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-24T16:38:13.261 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:13.260+0000 7ff572c0f180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-24T16:38:13.261 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:13.260+0000 7ff572c0f180 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: prometheus 2023-01-24T16:38:14.296 INFO:tasks.ceph.mgr.0.smithi097.stderr:2023-01-24T16:38:14.294+0000 7ff52c0dc700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-01-24T16:38:14.411 INFO:tasks.ceph.osd.0.smithi097.stderr:2023-01-24T16:38:14.409+0000 7faa7d6ce700 -1 osd.0 0 waiting for initial osdmap 2023-01-24T16:38:14.425 INFO:tasks.ceph.osd.0.smithi097.stderr:2023-01-24T16:38:14.423+0000 7faa764c4700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T16:38:14.852 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-01-24T16:38:14.850+0000 7f504084c700 -1 osd.2 0 waiting for initial osdmap 2023-01-24T16:38:14.855 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-01-24T16:38:14.854+0000 7f5039642700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T16:38:15.034 INFO:tasks.ceph.osd.1.smithi097.stderr:2023-01-24T16:38:15.032+0000 7fc0b79f3700 -1 osd.1 0 waiting for initial osdmap 2023-01-24T16:38:15.035 INFO:tasks.ceph.osd.1.smithi097.stderr:2023-01-24T16:38:15.034+0000 7fc0b07e9700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T16:38:15.896 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T16:38:16.199 INFO:teuthology.misc.health.smithi097.stdout: 2023-01-24T16:38:16.199 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":8,"fsid":"ba9bee0e-b74d-4668-a675-133390900940","created":"2023-01-24T16:38:07.892199+0000","modified":"2023-01-24T16:38:15.420733+0000","last_up_change":"2023-01-24T16:38:15.420733+0000","last_in_change":"2023-01-24T16:38:08.842585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a9a7d912-dc32-4e2c-a692-97890541722c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":79845},{"type":"v1","addr":"172.21.15.97:6809","nonce":79845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":79845},{"type":"v1","addr":"172.21.15.97:6811","nonce":79845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":79845},{"type":"v1","addr":"172.21.15.97:6815","nonce":79845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":79845},{"type":"v1","addr":"172.21.15.97:6813","nonce":79845}]},"public_addr":"172.21.15.97:6809/79845","cluster_addr":"172.21.15.97:6811/79845","heartbeat_back_addr":"172.21.15.97:6815/79845","heartbeat_front_addr":"172.21.15.97:6813/79845","state":["exists","up"]},{"osd":1,"uuid":"3719c1fb-f16b-4fa9-8110-43052f05c318","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":79846},{"type":"v1","addr":"172.21.15.97:6801","nonce":79846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":79846},{"type":"v1","addr":"172.21.15.97:6803","nonce":79846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":79846},{"type":"v1","addr":"172.21.15.97:6807","nonce":79846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":79846},{"type":"v1","addr":"172.21.15.97:6805","nonce":79846}]},"public_addr":"172.21.15.97:6801/79846","cluster_addr":"172.21.15.97:6803/79846","heartbeat_back_addr":"172.21.15.97:6807/79846","heartbeat_front_addr":"172.21.15.97:6805/79846","state":["exists","up"]},{"osd":2,"uuid":"3c859fd8-83e0-478d-8766-276c16357101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":79875},{"type":"v1","addr":"172.21.15.97:6817","nonce":79875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":79875},{"type":"v1","addr":"172.21.15.97:6819","nonce":79875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":79875},{"type":"v1","addr":"172.21.15.97:6823","nonce":79875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":79875},{"type":"v1","addr":"172.21.15.97:6821","nonce":79875}]},"public_addr":"172.21.15.97:6817/79875","cluster_addr":"172.21.15.97:6819/79875","heartbeat_back_addr":"172.21.15.97:6823/79875","heartbeat_front_addr":"172.21.15.97:6821/79875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T16:38:16.211 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-01-24T16:38:16.211 INFO:tasks.ceph:Creating RBD pool 2023-01-24T16:38:16.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd pool create rbd 8 2023-01-24T16:38:17.042 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'rbd' created 2023-01-24T16:38:17.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-01-24T16:38:18.048 INFO:teuthology.orchestra.run.smithi097.stderr:enabled application 'rbd' on pool 'rbd' 2023-01-24T16:38:18.063 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-01-24T16:38:18.063 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-01-24T16:38:18.064 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-01-24T16:38:18.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-01-24T16:38:18.065 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-01-24T16:38:18.414 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:18.427 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":4,"active_gid":4101,"active_name":"0","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":79507},{"type":"v1","addr":"172.21.15.97:6825","nonce":79507}]},"active_addr":"172.21.15.97:6825/79507","active_change":"2023-01-24T16:38:13.273529+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":false,"error_string":"No module named 'packaging'","module_options":{}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":580156856}]},{"addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":2223936379}]},{"addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":4077809543}]}]}} 2023-01-24T16:38:18.429 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-01-24T16:38:18.429 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-01-24T16:38:18.430 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T16:38:18.725 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:18.726 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":10,"fsid":"ba9bee0e-b74d-4668-a675-133390900940","created":"2023-01-24T16:38:07.892199+0000","modified":"2023-01-24T16:38:18.036163+0000","last_up_change":"2023-01-24T16:38:15.420733+0000","last_in_change":"2023-01-24T16:38:08.842585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T16:38:16.526984+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T16:38:17.311816+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"a9a7d912-dc32-4e2c-a692-97890541722c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":79845},{"type":"v1","addr":"172.21.15.97:6809","nonce":79845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":79845},{"type":"v1","addr":"172.21.15.97:6811","nonce":79845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":79845},{"type":"v1","addr":"172.21.15.97:6815","nonce":79845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":79845},{"type":"v1","addr":"172.21.15.97:6813","nonce":79845}]},"public_addr":"172.21.15.97:6809/79845","cluster_addr":"172.21.15.97:6811/79845","heartbeat_back_addr":"172.21.15.97:6815/79845","heartbeat_front_addr":"172.21.15.97:6813/79845","state":["exists","up"]},{"osd":1,"uuid":"3719c1fb-f16b-4fa9-8110-43052f05c318","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":79846},{"type":"v1","addr":"172.21.15.97:6801","nonce":79846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":79846},{"type":"v1","addr":"172.21.15.97:6803","nonce":79846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":79846},{"type":"v1","addr":"172.21.15.97:6807","nonce":79846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":79846},{"type":"v1","addr":"172.21.15.97:6805","nonce":79846}]},"public_addr":"172.21.15.97:6801/79846","cluster_addr":"172.21.15.97:6803/79846","heartbeat_back_addr":"172.21.15.97:6807/79846","heartbeat_front_addr":"172.21.15.97:6805/79846","state":["exists","up"]},{"osd":2,"uuid":"3c859fd8-83e0-478d-8766-276c16357101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":79875},{"type":"v1","addr":"172.21.15.97:6817","nonce":79875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":79875},{"type":"v1","addr":"172.21.15.97:6819","nonce":79875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":79875},{"type":"v1","addr":"172.21.15.97:6823","nonce":79875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":79875},{"type":"v1","addr":"172.21.15.97:6821","nonce":79875}]},"public_addr":"172.21.15.97:6817/79875","cluster_addr":"172.21.15.97:6819/79875","heartbeat_back_addr":"172.21.15.97:6823/79875","heartbeat_front_addr":"172.21.15.97:6821/79875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:38:13.270598+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:38:13.267327+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:38:13.210037+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T16:38:18.738 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-01-24T16:38:18.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T16:38:19.035 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:19.035 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":10,"fsid":"ba9bee0e-b74d-4668-a675-133390900940","created":"2023-01-24T16:38:07.892199+0000","modified":"2023-01-24T16:38:18.036163+0000","last_up_change":"2023-01-24T16:38:15.420733+0000","last_in_change":"2023-01-24T16:38:08.842585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T16:38:16.526984+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T16:38:17.311816+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"a9a7d912-dc32-4e2c-a692-97890541722c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":79845},{"type":"v1","addr":"172.21.15.97:6809","nonce":79845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":79845},{"type":"v1","addr":"172.21.15.97:6811","nonce":79845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":79845},{"type":"v1","addr":"172.21.15.97:6815","nonce":79845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":79845},{"type":"v1","addr":"172.21.15.97:6813","nonce":79845}]},"public_addr":"172.21.15.97:6809/79845","cluster_addr":"172.21.15.97:6811/79845","heartbeat_back_addr":"172.21.15.97:6815/79845","heartbeat_front_addr":"172.21.15.97:6813/79845","state":["exists","up"]},{"osd":1,"uuid":"3719c1fb-f16b-4fa9-8110-43052f05c318","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":79846},{"type":"v1","addr":"172.21.15.97:6801","nonce":79846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":79846},{"type":"v1","addr":"172.21.15.97:6803","nonce":79846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":79846},{"type":"v1","addr":"172.21.15.97:6807","nonce":79846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":79846},{"type":"v1","addr":"172.21.15.97:6805","nonce":79846}]},"public_addr":"172.21.15.97:6801/79846","cluster_addr":"172.21.15.97:6803/79846","heartbeat_back_addr":"172.21.15.97:6807/79846","heartbeat_front_addr":"172.21.15.97:6805/79846","state":["exists","up"]},{"osd":2,"uuid":"3c859fd8-83e0-478d-8766-276c16357101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":79875},{"type":"v1","addr":"172.21.15.97:6817","nonce":79875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":79875},{"type":"v1","addr":"172.21.15.97:6819","nonce":79875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":79875},{"type":"v1","addr":"172.21.15.97:6823","nonce":79875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":79875},{"type":"v1","addr":"172.21.15.97:6821","nonce":79875}]},"public_addr":"172.21.15.97:6817/79875","cluster_addr":"172.21.15.97:6819/79875","heartbeat_back_addr":"172.21.15.97:6823/79875","heartbeat_front_addr":"172.21.15.97:6821/79875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:38:13.270598+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:38:13.267327+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:38:13.210037+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T16:38:19.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-01-24T16:38:19.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-01-24T16:38:19.052 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-01-24T16:38:19.226 INFO:teuthology.orchestra.run.smithi097.stdout:34359738371 2023-01-24T16:38:19.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T16:38:19.232 INFO:teuthology.orchestra.run.smithi097.stdout:34359738371 2023-01-24T16:38:19.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T16:38:19.269 INFO:teuthology.orchestra.run.smithi097.stdout:34359738372 2023-01-24T16:38:19.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T16:38:19.629 INFO:teuthology.orchestra.run.smithi097.stdout:34359738370 2023-01-24T16:38:19.637 INFO:teuthology.orchestra.run.smithi097.stdout:34359738370 2023-01-24T16:38:19.643 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2023-01-24T16:38:19.651 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2023-01-24T16:38:19.678 INFO:teuthology.orchestra.run.smithi097.stdout:34359738371 2023-01-24T16:38:19.691 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2023-01-24T16:38:20.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T16:38:20.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T16:38:20.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T16:38:20.977 INFO:teuthology.orchestra.run.smithi097.stdout:34359738371 2023-01-24T16:38:20.991 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.2 2023-01-24T16:38:20.991 DEBUG:teuthology.parallel:result is None 2023-01-24T16:38:21.010 INFO:teuthology.orchestra.run.smithi097.stdout:34359738371 2023-01-24T16:38:21.023 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.1 2023-01-24T16:38:21.024 DEBUG:teuthology.parallel:result is None 2023-01-24T16:38:21.052 INFO:teuthology.orchestra.run.smithi097.stdout:34359738372 2023-01-24T16:38:21.065 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2023-01-24T16:38:21.066 DEBUG:teuthology.parallel:result is None 2023-01-24T16:38:21.066 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-24T16:38:21.066 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T16:38:21.402 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:21.404 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2023-01-24T16:38:21.415 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-01-24T16:38:21.280465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":22880,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":22592,"kb_avail":314549920,"statfs":{"total":322122547200,"available":322099118080,"internally_reserved":0,"allocated":270336,"data_stored":65088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23134208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000277"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.060762+0000","last_change":"2023-01-24T16:38:18.055933+0000","last_active":"2023-01-24T16:38:18.060762+0000","last_peered":"2023-01-24T16:38:18.060762+0000","last_clean":"2023-01-24T16:38:18.060762+0000","last_became_active":"2023-01-24T16:38:18.055695+0000","last_became_peered":"2023-01-24T16:38:18.055695+0000","last_unstale":"2023-01-24T16:38:18.060762+0000","last_undegraded":"2023-01-24T16:38:18.060762+0000","last_fullsized":"2023-01-24T16:38:18.060762+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T20:35:35.428594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.055399+0000","last_change":"2023-01-24T16:38:18.050837+0000","last_active":"2023-01-24T16:38:18.055399+0000","last_peered":"2023-01-24T16:38:18.055399+0000","last_clean":"2023-01-24T16:38:18.055399+0000","last_became_active":"2023-01-24T16:38:18.050561+0000","last_became_peered":"2023-01-24T16:38:18.050561+0000","last_unstale":"2023-01-24T16:38:18.055399+0000","last_undegraded":"2023-01-24T16:38:18.055399+0000","last_fullsized":"2023-01-24T16:38:18.055399+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T21:04:28.044363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.060807+0000","last_change":"2023-01-24T16:38:18.055657+0000","last_active":"2023-01-24T16:38:18.060807+0000","last_peered":"2023-01-24T16:38:18.060807+0000","last_clean":"2023-01-24T16:38:18.060807+0000","last_became_active":"2023-01-24T16:38:18.055403+0000","last_became_peered":"2023-01-24T16:38:18.055403+0000","last_unstale":"2023-01-24T16:38:18.060807+0000","last_undegraded":"2023-01-24T16:38:18.060807+0000","last_fullsized":"2023-01-24T16:38:18.060807+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T17:24:09.429151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.057955+0000","last_change":"2023-01-24T16:38:18.052879+0000","last_active":"2023-01-24T16:38:18.057955+0000","last_peered":"2023-01-24T16:38:18.057955+0000","last_clean":"2023-01-24T16:38:18.057955+0000","last_became_active":"2023-01-24T16:38:18.052611+0000","last_became_peered":"2023-01-24T16:38:18.052611+0000","last_unstale":"2023-01-24T16:38:18.057955+0000","last_undegraded":"2023-01-24T16:38:18.057955+0000","last_fullsized":"2023-01-24T16:38:18.057955+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T03:04:32.707715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.053751+0000","last_change":"2023-01-24T16:38:18.053751+0000","last_active":"2023-01-24T16:38:18.053751+0000","last_peered":"2023-01-24T16:38:18.053751+0000","last_clean":"2023-01-24T16:38:18.053751+0000","last_became_active":"2023-01-24T16:38:18.053485+0000","last_became_peered":"2023-01-24T16:38:18.053485+0000","last_unstale":"2023-01-24T16:38:18.053751+0000","last_undegraded":"2023-01-24T16:38:18.053751+0000","last_fullsized":"2023-01-24T16:38:18.053751+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T01:20:38.156435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'64","reported_seq":96,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T16:38:19.140989+0000","last_change":"2023-01-24T16:38:19.046896+0000","last_active":"2023-01-24T16:38:19.140989+0000","last_peered":"2023-01-24T16:38:19.140989+0000","last_clean":"2023-01-24T16:38:19.140989+0000","last_became_active":"2023-01-24T16:38:19.046246+0000","last_became_peered":"2023-01-24T16:38:19.046246+0000","last_unstale":"2023-01-24T16:38:19.140989+0000","last_undegraded":"2023-01-24T16:38:19.140989+0000","last_fullsized":"2023-01-24T16:38:19.140989+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:18.036163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:18.036163+0000","last_clean_scrub_stamp":"2023-01-24T16:38:18.036163+0000","objects_scrubbed":0,"log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T22:18:34.830963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.060774+0000","last_change":"2023-01-24T16:38:18.056354+0000","last_active":"2023-01-24T16:38:18.060774+0000","last_peered":"2023-01-24T16:38:18.060774+0000","last_clean":"2023-01-24T16:38:18.060774+0000","last_became_active":"2023-01-24T16:38:18.056206+0000","last_became_peered":"2023-01-24T16:38:18.056206+0000","last_unstale":"2023-01-24T16:38:18.060774+0000","last_undegraded":"2023-01-24T16:38:18.060774+0000","last_fullsized":"2023-01-24T16:38:18.060774+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T22:38:08.996396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.057972+0000","last_change":"2023-01-24T16:38:18.052908+0000","last_active":"2023-01-24T16:38:18.057972+0000","last_peered":"2023-01-24T16:38:18.057972+0000","last_clean":"2023-01-24T16:38:18.057972+0000","last_became_active":"2023-01-24T16:38:18.052631+0000","last_became_peered":"2023-01-24T16:38:18.052631+0000","last_unstale":"2023-01-24T16:38:18.057972+0000","last_undegraded":"2023-01-24T16:38:18.057972+0000","last_fullsized":"2023-01-24T16:38:18.057972+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T22:15:38.606918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.060773+0000","last_change":"2023-01-24T16:38:18.055630+0000","last_active":"2023-01-24T16:38:18.060773+0000","last_peered":"2023-01-24T16:38:18.060773+0000","last_clean":"2023-01-24T16:38:18.060773+0000","last_became_active":"2023-01-24T16:38:18.055389+0000","last_became_peered":"2023-01-24T16:38:18.055389+0000","last_unstale":"2023-01-24T16:38:18.060773+0000","last_undegraded":"2023-01-24T16:38:18.060773+0000","last_fullsized":"2023-01-24T16:38:18.060773+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T17:36:43.647551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7584,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850016,"statfs":{"total":107374182400,"available":107366416384,"internally_reserved":0,"allocated":90112,"data_stored":21696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7648,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849952,"statfs":{"total":107374182400,"available":107366350848,"internally_reserved":0,"allocated":90112,"data_stored":21696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7648,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849952,"statfs":{"total":107374182400,"available":107366350848,"internally_reserved":0,"allocated":90112,"data_stored":21696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T16:38:21.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T16:38:21.709 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:21.711 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2023-01-24T16:38:21.721 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-01-24T16:38:21.280465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":22880,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":22592,"kb_avail":314549920,"statfs":{"total":322122547200,"available":322099118080,"internally_reserved":0,"allocated":270336,"data_stored":65088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23134208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000277"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.060762+0000","last_change":"2023-01-24T16:38:18.055933+0000","last_active":"2023-01-24T16:38:18.060762+0000","last_peered":"2023-01-24T16:38:18.060762+0000","last_clean":"2023-01-24T16:38:18.060762+0000","last_became_active":"2023-01-24T16:38:18.055695+0000","last_became_peered":"2023-01-24T16:38:18.055695+0000","last_unstale":"2023-01-24T16:38:18.060762+0000","last_undegraded":"2023-01-24T16:38:18.060762+0000","last_fullsized":"2023-01-24T16:38:18.060762+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T20:35:35.428594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.055399+0000","last_change":"2023-01-24T16:38:18.050837+0000","last_active":"2023-01-24T16:38:18.055399+0000","last_peered":"2023-01-24T16:38:18.055399+0000","last_clean":"2023-01-24T16:38:18.055399+0000","last_became_active":"2023-01-24T16:38:18.050561+0000","last_became_peered":"2023-01-24T16:38:18.050561+0000","last_unstale":"2023-01-24T16:38:18.055399+0000","last_undegraded":"2023-01-24T16:38:18.055399+0000","last_fullsized":"2023-01-24T16:38:18.055399+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T21:04:28.044363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.060807+0000","last_change":"2023-01-24T16:38:18.055657+0000","last_active":"2023-01-24T16:38:18.060807+0000","last_peered":"2023-01-24T16:38:18.060807+0000","last_clean":"2023-01-24T16:38:18.060807+0000","last_became_active":"2023-01-24T16:38:18.055403+0000","last_became_peered":"2023-01-24T16:38:18.055403+0000","last_unstale":"2023-01-24T16:38:18.060807+0000","last_undegraded":"2023-01-24T16:38:18.060807+0000","last_fullsized":"2023-01-24T16:38:18.060807+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T17:24:09.429151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.057955+0000","last_change":"2023-01-24T16:38:18.052879+0000","last_active":"2023-01-24T16:38:18.057955+0000","last_peered":"2023-01-24T16:38:18.057955+0000","last_clean":"2023-01-24T16:38:18.057955+0000","last_became_active":"2023-01-24T16:38:18.052611+0000","last_became_peered":"2023-01-24T16:38:18.052611+0000","last_unstale":"2023-01-24T16:38:18.057955+0000","last_undegraded":"2023-01-24T16:38:18.057955+0000","last_fullsized":"2023-01-24T16:38:18.057955+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T03:04:32.707715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.053751+0000","last_change":"2023-01-24T16:38:18.053751+0000","last_active":"2023-01-24T16:38:18.053751+0000","last_peered":"2023-01-24T16:38:18.053751+0000","last_clean":"2023-01-24T16:38:18.053751+0000","last_became_active":"2023-01-24T16:38:18.053485+0000","last_became_peered":"2023-01-24T16:38:18.053485+0000","last_unstale":"2023-01-24T16:38:18.053751+0000","last_undegraded":"2023-01-24T16:38:18.053751+0000","last_fullsized":"2023-01-24T16:38:18.053751+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T01:20:38.156435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'64","reported_seq":96,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T16:38:19.140989+0000","last_change":"2023-01-24T16:38:19.046896+0000","last_active":"2023-01-24T16:38:19.140989+0000","last_peered":"2023-01-24T16:38:19.140989+0000","last_clean":"2023-01-24T16:38:19.140989+0000","last_became_active":"2023-01-24T16:38:19.046246+0000","last_became_peered":"2023-01-24T16:38:19.046246+0000","last_unstale":"2023-01-24T16:38:19.140989+0000","last_undegraded":"2023-01-24T16:38:19.140989+0000","last_fullsized":"2023-01-24T16:38:19.140989+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:18.036163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:18.036163+0000","last_clean_scrub_stamp":"2023-01-24T16:38:18.036163+0000","objects_scrubbed":0,"log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T22:18:34.830963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.060774+0000","last_change":"2023-01-24T16:38:18.056354+0000","last_active":"2023-01-24T16:38:18.060774+0000","last_peered":"2023-01-24T16:38:18.060774+0000","last_clean":"2023-01-24T16:38:18.060774+0000","last_became_active":"2023-01-24T16:38:18.056206+0000","last_became_peered":"2023-01-24T16:38:18.056206+0000","last_unstale":"2023-01-24T16:38:18.060774+0000","last_undegraded":"2023-01-24T16:38:18.060774+0000","last_fullsized":"2023-01-24T16:38:18.060774+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T22:38:08.996396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.057972+0000","last_change":"2023-01-24T16:38:18.052908+0000","last_active":"2023-01-24T16:38:18.057972+0000","last_peered":"2023-01-24T16:38:18.057972+0000","last_clean":"2023-01-24T16:38:18.057972+0000","last_became_active":"2023-01-24T16:38:18.052631+0000","last_became_peered":"2023-01-24T16:38:18.052631+0000","last_unstale":"2023-01-24T16:38:18.057972+0000","last_undegraded":"2023-01-24T16:38:18.057972+0000","last_fullsized":"2023-01-24T16:38:18.057972+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T22:15:38.606918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-01-24T16:38:18.060773+0000","last_change":"2023-01-24T16:38:18.055630+0000","last_active":"2023-01-24T16:38:18.060773+0000","last_peered":"2023-01-24T16:38:18.060773+0000","last_clean":"2023-01-24T16:38:18.060773+0000","last_became_active":"2023-01-24T16:38:18.055389+0000","last_became_peered":"2023-01-24T16:38:18.055389+0000","last_unstale":"2023-01-24T16:38:18.060773+0000","last_undegraded":"2023-01-24T16:38:18.060773+0000","last_fullsized":"2023-01-24T16:38:18.060773+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:38:17.035429+0000","last_clean_scrub_stamp":"2023-01-24T16:38:17.035429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T17:36:43.647551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7584,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850016,"statfs":{"total":107374182400,"available":107366416384,"internally_reserved":0,"allocated":90112,"data_stored":21696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7648,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849952,"statfs":{"total":107374182400,"available":107366350848,"internally_reserved":0,"allocated":90112,"data_stored":21696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7648,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849952,"statfs":{"total":107374182400,"available":107366350848,"internally_reserved":0,"allocated":90112,"data_stored":21696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T16:38:21.723 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-24T16:38:21.723 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-01-24T16:38:21.723 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-01-24T16:38:21.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-01-24T16:38:22.090 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:22.091 INFO:teuthology.orchestra.run.smithi097.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-01-24T16:38:22.104 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-01-24T16:38:22.105 INFO:teuthology.run_tasks:Running task rgw... 2023-01-24T16:38:22.124 DEBUG:tasks.rgw:config is {'client.0': None} 2023-01-24T16:38:22.125 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-01-24T16:38:22.126 INFO:tasks.rgw:Creating data pools 2023-01-24T16:38:22.126 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-01-24T16:38:22.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2023-01-24T16:38:23.284 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'default.rgw.buckets.data' created 2023-01-24T16:38:23.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-01-24T16:38:24.297 INFO:teuthology.orchestra.run.smithi097.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-01-24T16:38:24.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-01-24T16:38:25.297 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'default.rgw.buckets.index' created 2023-01-24T16:38:25.308 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-01-24T16:38:26.311 INFO:teuthology.orchestra.run.smithi097.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-01-24T16:38:26.321 DEBUG:tasks.rgw:Pools created 2023-01-24T16:38:26.322 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2023-01-24T16:38:26.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2023-01-24T16:38:26.323 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2023-01-24T16:38:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:38:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:38:26.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:26.393+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.307 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.306+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.307 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.306+0000 7f5db279f680 20 realm 2023-01-24T16:38:28.307 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.306+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.308+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.308+0000 7f5db279f680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T16:38:28.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.308+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.309+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.309+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.309+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.309+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.310+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.310+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.310+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.310+0000 7f5db279f680 10 Creating default zonegroup 2023-01-24T16:38:28.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.310+0000 7f5db279f680 20 get_system_obj_state: rctx=0x7fff10e97da0 obj=:.pools.avail state=0x55c222a98940 s->prefetch_data=0 2023-01-24T16:38:28.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.310+0000 7f5db279f680 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-01-24T16:38:28.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.320+0000 7f5db279f680 10 WARNING: store->list_zones() returned r=-2 2023-01-24T16:38:28.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.320+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.320+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.322+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.322+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.322+0000 7f5db279f680 10 could not read realm id: (2) No such file or directory 2023-01-24T16:38:28.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.322+0000 7f5db279f680 10 WARNING: failed to set zone as default, r=-22 2023-01-24T16:38:28.324 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.322+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.324 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.323+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.325+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.325+0000 7f5db279f680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T16:38:28.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.325+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.327 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.326+0000 7f5db279f680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T16:38:28.327 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.326+0000 7f5db279f680 10 Using default name default 2023-01-24T16:38:28.327 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.326+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.326+0000 7f5db279f680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:28.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.326+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.326+0000 7f5db279f680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T16:38:28.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.326+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.326+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.326+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.327+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.327+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.327+0000 7f5db279f680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:28.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.335+0000 7f5db279f680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:28.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.335+0000 7f5db279f680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:28.336 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.335+0000 7f5db279f680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:28.336 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.335+0000 7f5db279f680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:28.336 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.335+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.336 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.335+0000 7f5db279f680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:28.336 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.335+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.335+0000 7f5db279f680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T16:38:28.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.335+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:28.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T16:38:28.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 20 zone default found 2023-01-24T16:38:28.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 4 Realm: () 2023-01-24T16:38:28.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 4 ZoneGroup: default (2097a68d-863f-4181-8202-77da07845417) 2023-01-24T16:38:28.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 4 Zone: default (e9b1a570-bc22-4f42-b7c6-96dbb40f320d) 2023-01-24T16:38:28.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T16:38:28.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 20 zonegroup default 2023-01-24T16:38:28.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:28.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:28.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:28.336+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:30.324 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:30.323+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:30.324 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:30.323+0000 7f5db279f680 20 rados->read ofs=0 len=0 2023-01-24T16:38:30.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:30.323+0000 7f5db279f680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:30.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:30.323+0000 7f5db279f680 20 started sync module instance, tier type = 2023-01-24T16:38:30.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:30.323+0000 7f5db279f680 20 started zone id=e9b1a570-bc22-4f42-b7c6-96dbb40f320d (name=default) with tier type = 2023-01-24T16:38:32.352 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.351+0000 7f5db279f680 20 add_watcher() i=0 2023-01-24T16:38:32.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.351+0000 7f5db279f680 20 add_watcher() i=1 2023-01-24T16:38:32.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.351+0000 7f5db279f680 20 add_watcher() i=2 2023-01-24T16:38:32.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.351+0000 7f5db279f680 20 add_watcher() i=3 2023-01-24T16:38:32.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.351+0000 7f5db279f680 20 add_watcher() i=4 2023-01-24T16:38:32.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.351+0000 7f5db279f680 20 add_watcher() i=5 2023-01-24T16:38:32.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.351+0000 7f5db279f680 20 add_watcher() i=6 2023-01-24T16:38:32.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.352+0000 7f5db279f680 20 add_watcher() i=7 2023-01-24T16:38:32.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.352+0000 7f5db279f680 2 all 8 watchers are set, enabling cache 2023-01-24T16:38:32.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-01-24T16:38:32.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-01-24T16:38:32.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-01-24T16:38:32.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-01-24T16:38:32.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-01-24T16:38:32.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-01-24T16:38:32.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-01-24T16:38:32.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-01-24T16:38:32.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-01-24T16:38:32.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.4 2023-01-24T16:38:32.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.353+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.4 does not exist 2023-01-24T16:38:32.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.5 2023-01-24T16:38:32.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.5 does not exist 2023-01-24T16:38:32.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.6 2023-01-24T16:38:32.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.6 does not exist 2023-01-24T16:38:32.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.7 2023-01-24T16:38:32.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.7 does not exist 2023-01-24T16:38:32.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.8 2023-01-24T16:38:32.380 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.380 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.8 does not exist 2023-01-24T16:38:32.380 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.9 2023-01-24T16:38:32.380 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.354+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.355+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.9 does not exist 2023-01-24T16:38:32.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.355+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.10 2023-01-24T16:38:32.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.355+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.355+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.10 does not exist 2023-01-24T16:38:32.382 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.355+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.11 2023-01-24T16:38:32.382 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.355+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.382 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.355+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.11 does not exist 2023-01-24T16:38:32.382 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.355+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.12 2023-01-24T16:38:32.382 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.355+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.355+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.12 does not exist 2023-01-24T16:38:32.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.355+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.13 2023-01-24T16:38:32.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.355+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.13 does not exist 2023-01-24T16:38:32.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.14 2023-01-24T16:38:32.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.14 does not exist 2023-01-24T16:38:32.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.15 2023-01-24T16:38:32.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.15 does not exist 2023-01-24T16:38:32.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.16 2023-01-24T16:38:32.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.16 does not exist 2023-01-24T16:38:32.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.17 2023-01-24T16:38:32.386 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.386 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.17 does not exist 2023-01-24T16:38:32.386 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.18 2023-01-24T16:38:32.386 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.356+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.18 does not exist 2023-01-24T16:38:32.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.19 2023-01-24T16:38:32.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.19 does not exist 2023-01-24T16:38:32.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.20 2023-01-24T16:38:32.388 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.388 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.20 does not exist 2023-01-24T16:38:32.388 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.21 2023-01-24T16:38:32.388 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.388 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.21 does not exist 2023-01-24T16:38:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.22 2023-01-24T16:38:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.22 does not exist 2023-01-24T16:38:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.23 2023-01-24T16:38:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.357+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.390 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.23 does not exist 2023-01-24T16:38:32.390 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.24 2023-01-24T16:38:32.390 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.390 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.24 does not exist 2023-01-24T16:38:32.390 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.25 2023-01-24T16:38:32.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.25 does not exist 2023-01-24T16:38:32.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.26 2023-01-24T16:38:32.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.26 does not exist 2023-01-24T16:38:32.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.27 2023-01-24T16:38:32.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.27 does not exist 2023-01-24T16:38:32.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.28 2023-01-24T16:38:32.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.358+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.28 does not exist 2023-01-24T16:38:32.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.29 2023-01-24T16:38:32.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.29 does not exist 2023-01-24T16:38:32.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.30 2023-01-24T16:38:32.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.30 does not exist 2023-01-24T16:38:32.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.31 2023-01-24T16:38:32.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.31 does not exist 2023-01-24T16:38:32.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.32 2023-01-24T16:38:32.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.359+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.32 does not exist 2023-01-24T16:38:32.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.33 2023-01-24T16:38:32.396 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.396 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.33 does not exist 2023-01-24T16:38:32.396 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.34 2023-01-24T16:38:32.396 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.34 does not exist 2023-01-24T16:38:32.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.35 2023-01-24T16:38:32.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.35 does not exist 2023-01-24T16:38:32.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.36 2023-01-24T16:38:32.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.36 does not exist 2023-01-24T16:38:32.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.37 2023-01-24T16:38:32.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.360+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.37 does not exist 2023-01-24T16:38:32.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.38 2023-01-24T16:38:32.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.38 does not exist 2023-01-24T16:38:32.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.39 2023-01-24T16:38:32.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.400 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.39 does not exist 2023-01-24T16:38:32.400 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.40 2023-01-24T16:38:32.400 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.400 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.40 does not exist 2023-01-24T16:38:32.401 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.41 2023-01-24T16:38:32.401 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.401 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.41 does not exist 2023-01-24T16:38:32.401 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.42 2023-01-24T16:38:32.402 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.361+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.402 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.42 does not exist 2023-01-24T16:38:32.402 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.43 2023-01-24T16:38:32.402 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.43 does not exist 2023-01-24T16:38:32.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.44 2023-01-24T16:38:32.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.44 does not exist 2023-01-24T16:38:32.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.45 2023-01-24T16:38:32.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.45 does not exist 2023-01-24T16:38:32.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.46 2023-01-24T16:38:32.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.405 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.46 does not exist 2023-01-24T16:38:32.405 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.47 2023-01-24T16:38:32.405 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.362+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.405 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.47 does not exist 2023-01-24T16:38:32.405 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.48 2023-01-24T16:38:32.406 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.406 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.48 does not exist 2023-01-24T16:38:32.406 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.49 2023-01-24T16:38:32.406 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.407 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.49 does not exist 2023-01-24T16:38:32.407 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.50 2023-01-24T16:38:32.407 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.407 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.50 does not exist 2023-01-24T16:38:32.407 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.51 2023-01-24T16:38:32.408 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.408 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.51 does not exist 2023-01-24T16:38:32.408 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.52 2023-01-24T16:38:32.408 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.363+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.408 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.52 does not exist 2023-01-24T16:38:32.409 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.53 2023-01-24T16:38:32.409 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.409 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.53 does not exist 2023-01-24T16:38:32.409 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.54 2023-01-24T16:38:32.409 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.54 does not exist 2023-01-24T16:38:32.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.55 2023-01-24T16:38:32.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.55 does not exist 2023-01-24T16:38:32.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.56 2023-01-24T16:38:32.411 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.411 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.56 does not exist 2023-01-24T16:38:32.411 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.57 2023-01-24T16:38:32.411 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.364+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.411 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.57 does not exist 2023-01-24T16:38:32.412 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.58 2023-01-24T16:38:32.412 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.412 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.58 does not exist 2023-01-24T16:38:32.412 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.59 2023-01-24T16:38:32.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.59 does not exist 2023-01-24T16:38:32.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.60 2023-01-24T16:38:32.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.60 does not exist 2023-01-24T16:38:32.414 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.61 2023-01-24T16:38:32.414 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.414 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.61 does not exist 2023-01-24T16:38:32.414 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.62 2023-01-24T16:38:32.414 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.365+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.415 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.62 does not exist 2023-01-24T16:38:32.415 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.63 2023-01-24T16:38:32.415 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.415 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.63 does not exist 2023-01-24T16:38:32.415 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.64 2023-01-24T16:38:32.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.64 does not exist 2023-01-24T16:38:32.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.65 2023-01-24T16:38:32.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.65 does not exist 2023-01-24T16:38:32.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.66 2023-01-24T16:38:32.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.66 does not exist 2023-01-24T16:38:32.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.67 2023-01-24T16:38:32.418 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.418 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.67 does not exist 2023-01-24T16:38:32.418 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.68 2023-01-24T16:38:32.418 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.366+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.418 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.367+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.68 does not exist 2023-01-24T16:38:32.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.367+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.69 2023-01-24T16:38:32.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.367+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.367+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.69 does not exist 2023-01-24T16:38:32.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.367+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.70 2023-01-24T16:38:32.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.367+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.420 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.367+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.70 does not exist 2023-01-24T16:38:32.420 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.367+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.71 2023-01-24T16:38:32.420 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.367+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.420 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.367+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.71 does not exist 2023-01-24T16:38:32.420 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.367+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.72 2023-01-24T16:38:32.421 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.367+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.421 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.72 does not exist 2023-01-24T16:38:32.421 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.73 2023-01-24T16:38:32.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.73 does not exist 2023-01-24T16:38:32.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.74 2023-01-24T16:38:32.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.74 does not exist 2023-01-24T16:38:32.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.75 2023-01-24T16:38:32.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.75 does not exist 2023-01-24T16:38:32.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.76 2023-01-24T16:38:32.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.76 does not exist 2023-01-24T16:38:32.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.77 2023-01-24T16:38:32.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.368+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.77 does not exist 2023-01-24T16:38:32.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.78 2023-01-24T16:38:32.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.78 does not exist 2023-01-24T16:38:32.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.79 2023-01-24T16:38:32.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.79 does not exist 2023-01-24T16:38:32.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.80 2023-01-24T16:38:32.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.80 does not exist 2023-01-24T16:38:32.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.81 2023-01-24T16:38:32.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.81 does not exist 2023-01-24T16:38:32.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.82 2023-01-24T16:38:32.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.369+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.82 does not exist 2023-01-24T16:38:32.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.83 2023-01-24T16:38:32.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.83 does not exist 2023-01-24T16:38:32.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.84 2023-01-24T16:38:32.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.84 does not exist 2023-01-24T16:38:32.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.85 2023-01-24T16:38:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.85 does not exist 2023-01-24T16:38:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.86 2023-01-24T16:38:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.86 does not exist 2023-01-24T16:38:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.87 2023-01-24T16:38:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.370+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.87 does not exist 2023-01-24T16:38:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.88 2023-01-24T16:38:32.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.88 does not exist 2023-01-24T16:38:32.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.89 2023-01-24T16:38:32.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.89 does not exist 2023-01-24T16:38:32.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.90 2023-01-24T16:38:32.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.90 does not exist 2023-01-24T16:38:32.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.91 2023-01-24T16:38:32.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.91 does not exist 2023-01-24T16:38:32.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.92 2023-01-24T16:38:32.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.92 does not exist 2023-01-24T16:38:32.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.93 2023-01-24T16:38:32.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.371+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.93 does not exist 2023-01-24T16:38:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.94 2023-01-24T16:38:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.94 does not exist 2023-01-24T16:38:32.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.95 2023-01-24T16:38:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.95 does not exist 2023-01-24T16:38:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.96 2023-01-24T16:38:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.96 does not exist 2023-01-24T16:38:32.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.97 2023-01-24T16:38:32.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.443 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.97 does not exist 2023-01-24T16:38:32.443 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.98 2023-01-24T16:38:32.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.372+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.98 does not exist 2023-01-24T16:38:32.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.99 2023-01-24T16:38:32.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.99 does not exist 2023-01-24T16:38:32.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.100 2023-01-24T16:38:32.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.100 does not exist 2023-01-24T16:38:32.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.101 2023-01-24T16:38:32.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.101 does not exist 2023-01-24T16:38:32.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.102 2023-01-24T16:38:32.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.102 does not exist 2023-01-24T16:38:32.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.103 2023-01-24T16:38:32.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.373+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.103 does not exist 2023-01-24T16:38:32.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.104 2023-01-24T16:38:32.448 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.448 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.104 does not exist 2023-01-24T16:38:32.448 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.105 2023-01-24T16:38:32.448 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.105 does not exist 2023-01-24T16:38:32.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.106 2023-01-24T16:38:32.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.106 does not exist 2023-01-24T16:38:32.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.107 2023-01-24T16:38:32.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.107 does not exist 2023-01-24T16:38:32.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.108 2023-01-24T16:38:32.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.374+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.108 does not exist 2023-01-24T16:38:32.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.109 2023-01-24T16:38:32.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.109 does not exist 2023-01-24T16:38:32.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.110 2023-01-24T16:38:32.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.110 does not exist 2023-01-24T16:38:32.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.111 2023-01-24T16:38:32.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.111 does not exist 2023-01-24T16:38:32.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.112 2023-01-24T16:38:32.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.112 does not exist 2023-01-24T16:38:32.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.113 2023-01-24T16:38:32.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.113 does not exist 2023-01-24T16:38:32.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.114 2023-01-24T16:38:32.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.375+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.114 does not exist 2023-01-24T16:38:32.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.115 2023-01-24T16:38:32.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.115 does not exist 2023-01-24T16:38:32.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.116 2023-01-24T16:38:32.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.116 does not exist 2023-01-24T16:38:32.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.117 2023-01-24T16:38:32.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.117 does not exist 2023-01-24T16:38:32.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.118 2023-01-24T16:38:32.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.118 does not exist 2023-01-24T16:38:32.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.119 2023-01-24T16:38:32.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.376+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.119 does not exist 2023-01-24T16:38:32.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.120 2023-01-24T16:38:32.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.120 does not exist 2023-01-24T16:38:32.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.121 2023-01-24T16:38:32.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.121 does not exist 2023-01-24T16:38:32.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.122 2023-01-24T16:38:32.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.122 does not exist 2023-01-24T16:38:32.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.123 2023-01-24T16:38:32.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.123 does not exist 2023-01-24T16:38:32.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.124 2023-01-24T16:38:32.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.377+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.378+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.124 does not exist 2023-01-24T16:38:32.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.378+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.125 2023-01-24T16:38:32.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.378+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.378+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.125 does not exist 2023-01-24T16:38:32.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.378+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.126 2023-01-24T16:38:32.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.378+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.378+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.126 does not exist 2023-01-24T16:38:32.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.378+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.127 2023-01-24T16:38:32.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.378+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.378+0000 7f5db279f680 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.127 does not exist 2023-01-24T16:38:32.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.378+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2023-01-24T16:38:32.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.379+0000 7f5db279f680 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2023-01-24T16:38:32.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.381+0000 7f5d777de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:32.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.382+0000 7f5db279f680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:38:32.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:32.382+0000 7f5db279f680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:38:35.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.322+0000 7f5db279f680 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-01-24T16:38:35.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.322+0000 7f5db279f680 5 note: GC not initialized 2023-01-24T16:38:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.324+0000 7f5cdbfff700 20 reqs_thread_entry: start 2023-01-24T16:38:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.324+0000 7f5db279f680 20 init_complete bucket index max shards: 11 2023-01-24T16:38:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.324+0000 7f5ce67fc700 20 reqs_thread_entry: start 2023-01-24T16:38:35.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.324+0000 7f5db279f680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T16:38:35.327 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.324+0000 7f5cda7fc700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 16:39:05 2023 2023-01-24T16:38:35.327 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:38:35.327 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.324+0000 7f5cd9ffb700 20 reqs_thread_entry: start 2023-01-24T16:38:35.346 INFO:teuthology.orchestra.run.smithi097.stdout:[] 2023-01-24T16:38:35.347 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.345+0000 7f5db279f680 20 remove_watcher() i=0 2023-01-24T16:38:35.347 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.345+0000 7f5db279f680 2 removed watcher, disabling cache 2023-01-24T16:38:35.347 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.346+0000 7f5db279f680 20 remove_watcher() i=1 2023-01-24T16:38:35.350 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.349+0000 7f5db279f680 20 remove_watcher() i=2 2023-01-24T16:38:35.352 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.352+0000 7f5db279f680 20 remove_watcher() i=3 2023-01-24T16:38:35.355 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.355+0000 7f5db279f680 20 remove_watcher() i=4 2023-01-24T16:38:35.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.358+0000 7f5db279f680 20 remove_watcher() i=5 2023-01-24T16:38:35.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.361+0000 7f5db279f680 20 remove_watcher() i=6 2023-01-24T16:38:35.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.362+0000 7f5db279f680 20 remove_watcher() i=7 2023-01-24T16:38:35.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.363+0000 7f5d777de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:35.368 DEBUG:tasks.util.rgw: json result: [] 2023-01-24T16:38:35.368 INFO:tasks.rgw:Configuring storage class type = LUKEWARM 2023-01-24T16:38:35.369 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2023-01-24T16:38:35.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2023-01-24T16:38:35.369 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM 2023-01-24T16:38:35.416 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:38:35.416 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:38:35.421 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.421+0000 7f8b1f4fb680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:38:35.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.421+0000 7f8b1f4fb680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:38:35.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.424+0000 7f8a5bfff700 20 reqs_thread_entry: start 2023-01-24T16:38:35.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.424+0000 7f8b1f4fb680 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2023-01-24T16:38:35.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.424+0000 7f8b1f4fb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.425+0000 7f8b1f4fb680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:35.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.425+0000 7f8b1f4fb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.425+0000 7f8b1f4fb680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T16:38:35.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.433+0000 7f8b1f4fb680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:35.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.433+0000 7f8b1f4fb680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:35.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.433+0000 7f8b1f4fb680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:35.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.433+0000 7f8b1f4fb680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:35.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.433+0000 7f8b1f4fb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.433+0000 7f8b1f4fb680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:35.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.433+0000 7f8b1f4fb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.433+0000 7f8b1f4fb680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:35.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.433+0000 7f8b1f4fb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.433+0000 7f8b1f4fb680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T16:38:35.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.433+0000 7f8b1f4fb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.434+0000 7f8b1f4fb680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T16:38:35.438 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2023-01-24T16:38:35.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2023-01-24T16:38:35.438 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "default-placement", 2023-01-24T16:38:35.438 INFO:teuthology.orchestra.run.smithi097.stdout: "val": { 2023-01-24T16:38:35.439 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default-placement", 2023-01-24T16:38:35.439 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2023-01-24T16:38:35.439 INFO:teuthology.orchestra.run.smithi097.stdout: "storage_classes": [ 2023-01-24T16:38:35.439 INFO:teuthology.orchestra.run.smithi097.stdout: "LUKEWARM", 2023-01-24T16:38:35.440 INFO:teuthology.orchestra.run.smithi097.stdout: "STANDARD" 2023-01-24T16:38:35.440 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2023-01-24T16:38:35.440 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:35.440 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:35.440 INFO:teuthology.orchestra.run.smithi097.stdout:] 2023-01-24T16:38:35.441 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['LUKEWARM', 'STANDARD']}}] 2023-01-24T16:38:35.442 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2023-01-24T16:38:35.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2023-01-24T16:38:35.442 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2023-01-24T16:38:35.488 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:38:35.488 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:38:35.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.493+0000 7f9a78928680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:38:35.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.493+0000 7f9a78928680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:38:35.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.496+0000 7f99e57fa700 20 reqs_thread_entry: start 2023-01-24T16:38:35.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.496+0000 7f9a78928680 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2023-01-24T16:38:35.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.496+0000 7f9a78928680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.496+0000 7f9a78928680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:35.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.496+0000 7f9a78928680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.497+0000 7f9a78928680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T16:38:35.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.504+0000 7f9a78928680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:35.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.504+0000 7f9a78928680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:35.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.504+0000 7f9a78928680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:35.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.504+0000 7f9a78928680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:35.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.504+0000 7f9a78928680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.504+0000 7f9a78928680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T16:38:35.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.504+0000 7f9a78928680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.504+0000 7f9a78928680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:35.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.504+0000 7f9a78928680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.504+0000 7f9a78928680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:35.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.504+0000 7f9a78928680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.507+0000 7f9a78928680 20 rados_obj.operate() r=0 bl.length=386 2023-01-24T16:38:35.511 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-01-24T16:38:35.512 INFO:teuthology.orchestra.run.smithi097.stdout: "id": "e9b1a570-bc22-4f42-b7c6-96dbb40f320d", 2023-01-24T16:38:35.512 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default", 2023-01-24T16:38:35.512 INFO:teuthology.orchestra.run.smithi097.stdout: "domain_root": "default.rgw.meta:root", 2023-01-24T16:38:35.512 INFO:teuthology.orchestra.run.smithi097.stdout: "control_pool": "default.rgw.control", 2023-01-24T16:38:35.513 INFO:teuthology.orchestra.run.smithi097.stdout: "gc_pool": "default.rgw.log:gc", 2023-01-24T16:38:35.513 INFO:teuthology.orchestra.run.smithi097.stdout: "lc_pool": "default.rgw.log:lc", 2023-01-24T16:38:35.513 INFO:teuthology.orchestra.run.smithi097.stdout: "log_pool": "default.rgw.log", 2023-01-24T16:38:35.513 INFO:teuthology.orchestra.run.smithi097.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-01-24T16:38:35.513 INFO:teuthology.orchestra.run.smithi097.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-01-24T16:38:35.514 INFO:teuthology.orchestra.run.smithi097.stdout: "roles_pool": "default.rgw.meta:roles", 2023-01-24T16:38:35.514 INFO:teuthology.orchestra.run.smithi097.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-01-24T16:38:35.514 INFO:teuthology.orchestra.run.smithi097.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-01-24T16:38:35.515 INFO:teuthology.orchestra.run.smithi097.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-01-24T16:38:35.515 INFO:teuthology.orchestra.run.smithi097.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-01-24T16:38:35.515 INFO:teuthology.orchestra.run.smithi097.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-01-24T16:38:35.516 INFO:teuthology.orchestra.run.smithi097.stdout: "otp_pool": "default.rgw.otp", 2023-01-24T16:38:35.516 INFO:teuthology.orchestra.run.smithi097.stdout: "system_key": { 2023-01-24T16:38:35.516 INFO:teuthology.orchestra.run.smithi097.stdout: "access_key": "", 2023-01-24T16:38:35.516 INFO:teuthology.orchestra.run.smithi097.stdout: "secret_key": "" 2023-01-24T16:38:35.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:35.517 INFO:teuthology.orchestra.run.smithi097.stdout: "placement_pools": [ 2023-01-24T16:38:35.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2023-01-24T16:38:35.517 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "default-placement", 2023-01-24T16:38:35.517 INFO:teuthology.orchestra.run.smithi097.stdout: "val": { 2023-01-24T16:38:35.517 INFO:teuthology.orchestra.run.smithi097.stdout: "index_pool": "default.rgw.buckets.index", 2023-01-24T16:38:35.518 INFO:teuthology.orchestra.run.smithi097.stdout: "storage_classes": { 2023-01-24T16:38:35.518 INFO:teuthology.orchestra.run.smithi097.stdout: "LUKEWARM": { 2023-01-24T16:38:35.518 INFO:teuthology.orchestra.run.smithi097.stdout: "data_pool": "default.rgw.buckets.data.lukewarm" 2023-01-24T16:38:35.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:35.518 INFO:teuthology.orchestra.run.smithi097.stdout: "STANDARD": { 2023-01-24T16:38:35.519 INFO:teuthology.orchestra.run.smithi097.stdout: "data_pool": "default.rgw.buckets.data" 2023-01-24T16:38:35.519 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:35.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:35.519 INFO:teuthology.orchestra.run.smithi097.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-01-24T16:38:35.519 INFO:teuthology.orchestra.run.smithi097.stdout: "index_type": 0 2023-01-24T16:38:35.519 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:35.520 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:35.520 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2023-01-24T16:38:35.520 INFO:teuthology.orchestra.run.smithi097.stdout: "realm_id": "", 2023-01-24T16:38:35.520 INFO:teuthology.orchestra.run.smithi097.stdout: "notif_pool": "default.rgw.log:notif" 2023-01-24T16:38:35.520 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-01-24T16:38:35.521 DEBUG:tasks.util.rgw: json result: {'id': 'e9b1a570-bc22-4f42-b7c6-96dbb40f320d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-01-24T16:38:35.522 INFO:tasks.rgw:Configuring storage class type = FROZEN 2023-01-24T16:38:35.522 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2023-01-24T16:38:35.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2023-01-24T16:38:35.523 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN 2023-01-24T16:38:35.568 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:38:35.568 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:38:35.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.573+0000 7f8499f32680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:38:35.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.573+0000 7f8499f32680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:38:35.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.576+0000 7f83d67fc700 20 reqs_thread_entry: start 2023-01-24T16:38:35.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.576+0000 7f8499f32680 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2023-01-24T16:38:35.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.576+0000 7f8499f32680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.578 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.577+0000 7f8499f32680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:35.578 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.577+0000 7f8499f32680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.580+0000 7f8499f32680 20 rados_obj.operate() r=0 bl.length=940 2023-01-24T16:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.586+0000 7f8499f32680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.586+0000 7f8499f32680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.586+0000 7f8499f32680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:35.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.586+0000 7f8499f32680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.587+0000 7f8499f32680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.587+0000 7f8499f32680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.587+0000 7f8499f32680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.587+0000 7f8499f32680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:35.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.587+0000 7f8499f32680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.587+0000 7f8499f32680 20 rados_obj.operate() r=0 bl.length=386 2023-01-24T16:38:35.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.587+0000 7f8499f32680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.587+0000 7f8499f32680 20 rados_obj.operate() r=0 bl.length=940 2023-01-24T16:38:35.591 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2023-01-24T16:38:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2023-01-24T16:38:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "default-placement", 2023-01-24T16:38:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "val": { 2023-01-24T16:38:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default-placement", 2023-01-24T16:38:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2023-01-24T16:38:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "storage_classes": [ 2023-01-24T16:38:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "FROZEN", 2023-01-24T16:38:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: "LUKEWARM", 2023-01-24T16:38:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: "STANDARD" 2023-01-24T16:38:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2023-01-24T16:38:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:35.594 INFO:teuthology.orchestra.run.smithi097.stdout:] 2023-01-24T16:38:35.595 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2023-01-24T16:38:35.595 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2023-01-24T16:38:35.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2023-01-24T16:38:35.596 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2023-01-24T16:38:35.642 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:38:35.642 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:38:35.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.646+0000 7f35d3487680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:38:35.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.646+0000 7f35d3487680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:38:35.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.650+0000 7f350ffff700 20 reqs_thread_entry: start 2023-01-24T16:38:35.651 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.650+0000 7f35d3487680 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2023-01-24T16:38:35.651 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.650+0000 7f35d3487680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.652 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.650+0000 7f35d3487680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:35.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.650+0000 7f35d3487680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.651+0000 7f35d3487680 20 rados_obj.operate() r=0 bl.length=940 2023-01-24T16:38:35.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.658+0000 7f35d3487680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:35.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.658+0000 7f35d3487680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:35.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.658+0000 7f35d3487680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:35.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.658+0000 7f35d3487680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:35.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.658+0000 7f35d3487680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.658+0000 7f35d3487680 20 rados_obj.operate() r=0 bl.length=940 2023-01-24T16:38:35.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.658+0000 7f35d3487680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.658+0000 7f35d3487680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:35.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.658+0000 7f35d3487680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.658+0000 7f35d3487680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:35.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.658+0000 7f35d3487680 20 rados->read ofs=0 len=0 2023-01-24T16:38:35.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:35.659+0000 7f35d3487680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:35.663 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-01-24T16:38:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: "id": "e9b1a570-bc22-4f42-b7c6-96dbb40f320d", 2023-01-24T16:38:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default", 2023-01-24T16:38:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: "domain_root": "default.rgw.meta:root", 2023-01-24T16:38:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: "control_pool": "default.rgw.control", 2023-01-24T16:38:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: "gc_pool": "default.rgw.log:gc", 2023-01-24T16:38:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: "lc_pool": "default.rgw.log:lc", 2023-01-24T16:38:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: "log_pool": "default.rgw.log", 2023-01-24T16:38:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-01-24T16:38:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-01-24T16:38:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: "roles_pool": "default.rgw.meta:roles", 2023-01-24T16:38:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-01-24T16:38:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-01-24T16:38:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-01-24T16:38:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-01-24T16:38:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-01-24T16:38:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: "otp_pool": "default.rgw.otp", 2023-01-24T16:38:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: "system_key": { 2023-01-24T16:38:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: "access_key": "", 2023-01-24T16:38:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: "secret_key": "" 2023-01-24T16:38:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: "placement_pools": [ 2023-01-24T16:38:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2023-01-24T16:38:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "default-placement", 2023-01-24T16:38:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: "val": { 2023-01-24T16:38:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: "index_pool": "default.rgw.buckets.index", 2023-01-24T16:38:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: "storage_classes": { 2023-01-24T16:38:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: "FROZEN": { 2023-01-24T16:38:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: "data_pool": "default.rgw.buckets.data.frozen" 2023-01-24T16:38:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: "LUKEWARM": { 2023-01-24T16:38:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: "data_pool": "default.rgw.buckets.data.lukewarm" 2023-01-24T16:38:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: "STANDARD": { 2023-01-24T16:38:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: "data_pool": "default.rgw.buckets.data" 2023-01-24T16:38:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-01-24T16:38:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "index_type": 0 2023-01-24T16:38:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2023-01-24T16:38:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: "realm_id": "", 2023-01-24T16:38:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: "notif_pool": "default.rgw.log:notif" 2023-01-24T16:38:35.678 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-01-24T16:38:35.680 DEBUG:tasks.util.rgw: json result: {'id': 'e9b1a570-bc22-4f42-b7c6-96dbb40f320d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-01-24T16:38:35.680 INFO:tasks.rgw:Starting rgw... 2023-01-24T16:38:35.681 INFO:tasks.rgw:rgw client.0 config is {} 2023-01-24T16:38:35.681 INFO:tasks.rgw:Using beast as radosgw frontend 2023-01-24T16:38:35.682 INFO:tasks.rgw.client.0:Restarting daemon 2023-01-24T16:38:35.682 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-01-24T16:38:35.685 INFO:tasks.rgw.client.0:Started 2023-01-24T16:38:35.685 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi097.front.sepia.ceph.com:80/ 2023-01-24T16:38:35.686 DEBUG:teuthology.orchestra.run.smithi097:> curl http://smithi097.front.sepia.ceph.com:80/ 2023-01-24T16:38:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T16:38:35.717 INFO:teuthology.orchestra.run.smithi097.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T16:38:35.718 INFO:teuthology.orchestra.run.smithi097.stderr: Dload Upload Total Spent Left Speed 2023-01-24T16:38:35.718 INFO:teuthology.orchestra.run.smithi097.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi097.front.sepia.ceph.com port 80: Connection refused 2023-01-24T16:38:36.720 DEBUG:teuthology.orchestra.run.smithi097:> curl http://smithi097.front.sepia.ceph.com:80/ 2023-01-24T16:38:36.740 INFO:teuthology.orchestra.run.smithi097.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T16:38:36.741 INFO:teuthology.orchestra.run.smithi097.stderr: Dload Upload Total Spent Left Speed 2023-01-24T16:38:36.741 INFO:teuthology.orchestra.run.smithi097.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2023-01-24T16:38:36.742 INFO:teuthology.orchestra.run.smithi097.stdout:anonymous 2023-01-24T16:38:36.743 INFO:teuthology.run_tasks:Running task s3tests... 2023-01-24T16:38:36.757 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'extra_attrs': ['lifecycle'], 'force-branch': 'ceph-quincy', 'lc_debug_interval': 10, 'rgw_server': 'client.0', 'storage classes': 'LUKEWARM, FROZEN'}} 2023-01-24T16:38:36.758 INFO:tasks.s3tests:Downloading s3-tests... 2023-01-24T16:38:36.758 INFO:tasks.s3tests:Using branch 'ceph-quincy' for s3tests 2023-01-24T16:38:36.758 DEBUG:teuthology.orchestra.run.smithi097:> git clone -b ceph-quincy https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests 2023-01-24T16:38:36.797 INFO:teuthology.orchestra.run.smithi097.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests'... 2023-01-24T16:38:37.529 INFO:tasks.s3tests:Creating rgw users... 2023-01-24T16:38:37.530 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2023-01-24T16:38:37.531 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key BNAVUMGDINJPSXUTEDAY --secret R66pXyd6bdS8/nlLqk3nvlGoVoVS3Vfg4JaBpajzSgm6vRF3gOwOgQ== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-01-24T16:38:37.620 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:38:37.621 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:38:37.625 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.625+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.627 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.626+0000 7f25cc38c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.627 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.626+0000 7f25cc38c680 20 realm 2023-01-24T16:38:37.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.626+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.626+0000 7f25cc38c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.626+0000 7f25cc38c680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T16:38:37.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.626+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.629 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.626+0000 7f25cc38c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.629 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.626+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.629 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.627+0000 7f25cc38c680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:37.629 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.627+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.629 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.627+0000 7f25cc38c680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:38:37.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.627+0000 7f25cc38c680 20 searching for the correct realm 2023-01-24T16:38:37.635 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.634+0000 7f25cc38c680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:37.635 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.634+0000 7f25cc38c680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:37.636 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.634+0000 7f25cc38c680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:37.636 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.634+0000 7f25cc38c680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:37.636 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.634+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.636 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.634+0000 7f25cc38c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.634+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.634+0000 7f25cc38c680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:37.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.634+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.634+0000 7f25cc38c680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:37.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.634+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.635+0000 7f25cc38c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.635+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.635+0000 7f25cc38c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.635+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.635+0000 7f25cc38c680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:37.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.641+0000 7f25cc38c680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:37.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.641+0000 7f25cc38c680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:37.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.641+0000 7f25cc38c680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:37.644 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.641+0000 7f25cc38c680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:37.644 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.641+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.644 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.641+0000 7f25cc38c680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:37.644 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.641+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.645 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.641+0000 7f25cc38c680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:38:37.645 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.641+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.645 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.641+0000 7f25cc38c680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:37.645 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.641+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.645 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:37.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 zone default found 2023-01-24T16:38:37.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 4 Realm: () 2023-01-24T16:38:37.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 4 ZoneGroup: default (2097a68d-863f-4181-8202-77da07845417) 2023-01-24T16:38:37.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 4 Zone: default (e9b1a570-bc22-4f42-b7c6-96dbb40f320d) 2023-01-24T16:38:37.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T16:38:37.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 zonegroup default 2023-01-24T16:38:37.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 started sync module instance, tier type = 2023-01-24T16:38:37.649 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.642+0000 7f25cc38c680 20 started zone id=e9b1a570-bc22-4f42-b7c6-96dbb40f320d (name=default) with tier type = 2023-01-24T16:38:37.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.662+0000 7f25cc38c680 20 add_watcher() i=0 2023-01-24T16:38:37.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.662+0000 7f25cc38c680 20 add_watcher() i=1 2023-01-24T16:38:37.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.662+0000 7f25cc38c680 20 add_watcher() i=2 2023-01-24T16:38:37.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.662+0000 7f25cc38c680 20 add_watcher() i=3 2023-01-24T16:38:37.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.662+0000 7f25cc38c680 20 add_watcher() i=4 2023-01-24T16:38:37.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.662+0000 7f25cc38c680 20 add_watcher() i=5 2023-01-24T16:38:37.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.662+0000 7f25cc38c680 20 add_watcher() i=6 2023-01-24T16:38:37.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.664+0000 7f25cc38c680 20 add_watcher() i=7 2023-01-24T16:38:37.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.664+0000 7f25cc38c680 2 all 8 watchers are set, enabling cache 2023-01-24T16:38:37.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.666+0000 7f250bfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:37.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.666+0000 7f25cc38c680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:38:37.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.666+0000 7f25cc38c680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:38:37.670 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.669+0000 7f25cc38c680 5 note: GC not initialized 2023-01-24T16:38:37.672 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.670+0000 7f2502ffd700 20 reqs_thread_entry: start 2023-01-24T16:38:37.672 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.670+0000 7f2500ff9700 20 reqs_thread_entry: start 2023-01-24T16:38:37.672 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.670+0000 7f25cc38c680 20 init_complete bucket index max shards: 11 2023-01-24T16:38:37.672 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.670+0000 7f25cc38c680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T16:38:37.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.670+0000 7f24eeffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 16:39:08 2023 2023-01-24T16:38:37.673 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:38:37.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.670+0000 7f24ee7fc700 20 reqs_thread_entry: start 2023-01-24T16:38:37.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.670+0000 7f25cc38c680 20 get_system_obj_state: rctx=0x7ffe8e474c68 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55cfec3ef940 s->prefetch_data=0 2023-01-24T16:38:37.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.670+0000 7f25cc38c680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-01-24T16:38:37.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2023-01-24T16:38:37.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T16:38:37.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 20 get_system_obj_state: rctx=0x7ffe8e474958 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x55cfec3ef940 s->prefetch_data=0 2023-01-24T16:38:37.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2023-01-24T16:38:37.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2023-01-24T16:38:37.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-01-24T16:38:37.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 20 get_system_obj_state: rctx=0x7ffe8e474958 obj=default.rgw.meta:users.keys:BNAVUMGDINJPSXUTEDAY state=0x55cfec3ef940 s->prefetch_data=0 2023-01-24T16:38:37.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 10 cache get: name=default.rgw.meta+users.keys+BNAVUMGDINJPSXUTEDAY : miss 2023-01-24T16:38:37.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 10 cache put: name=default.rgw.meta+users.keys+BNAVUMGDINJPSXUTEDAY info.flags=0x0 2023-01-24T16:38:37.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 10 adding default.rgw.meta+users.keys+BNAVUMGDINJPSXUTEDAY to cache LRU end 2023-01-24T16:38:37.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 20 get_system_obj_state: rctx=0x7ffe8e4741e8 obj=default.rgw.meta:users.keys:BNAVUMGDINJPSXUTEDAY state=0x55cfec3ef940 s->prefetch_data=0 2023-01-24T16:38:37.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 10 cache get: name=default.rgw.meta+users.keys+BNAVUMGDINJPSXUTEDAY : hit (negative entry) 2023-01-24T16:38:37.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 20 get_system_obj_state: rctx=0x7ffe8e474898 obj=default.rgw.meta:users.keys:BNAVUMGDINJPSXUTEDAY state=0x55cfec3ef940 s->prefetch_data=0 2023-01-24T16:38:37.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.671+0000 7f25cc38c680 10 cache get: name=default.rgw.meta+users.keys+BNAVUMGDINJPSXUTEDAY : hit (negative entry) 2023-01-24T16:38:37.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.672+0000 7f25cc38c680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T16:38:37.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.672+0000 7f25cc38c680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T16:38:37.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.672+0000 7f25cc38c680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-01-24T16:38:37.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.673+0000 7f25bd48a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 94351511742576 notifier 4239 bl.length()=640 2023-01-24T16:38:37.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.673+0000 7f25bd48a700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T16:38:37.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.673+0000 7f25bd48a700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T16:38:37.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.674+0000 7f25cc38c680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-01-24T16:38:37.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.674+0000 7f25cc38c680 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-01-24T16:38:37.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.674+0000 7f25cc38c680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-01-24T16:38:37.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.674+0000 7f25bd48a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 94351511764304 notifier 4239 bl.length()=194 2023-01-24T16:38:37.680 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.674+0000 7f25bd48a700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-01-24T16:38:37.680 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.674+0000 7f25bd48a700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-01-24T16:38:37.680 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.675+0000 7f25cc38c680 10 cache put: name=default.rgw.meta+users.keys+BNAVUMGDINJPSXUTEDAY info.flags=0x7 2023-01-24T16:38:37.680 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.675+0000 7f25cc38c680 10 moving default.rgw.meta+users.keys+BNAVUMGDINJPSXUTEDAY to cache LRU end 2023-01-24T16:38:37.680 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.675+0000 7f25cc38c680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.keys:BNAVUMGDINJPSXUTEDAY, ofs0, ns] 2023-01-24T16:38:37.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.676+0000 7f25bd48a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182401 cookie 94351511765408 notifier 4239 bl.length()=186 2023-01-24T16:38:37.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.676+0000 7f25bd48a700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+BNAVUMGDINJPSXUTEDAY info.flags=0x7 2023-01-24T16:38:37.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.676+0000 7f25bd48a700 10 rgw watcher librados: moving default.rgw.meta+users.keys+BNAVUMGDINJPSXUTEDAY to cache LRU end 2023-01-24T16:38:37.681 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-01-24T16:38:37.682 INFO:teuthology.orchestra.run.smithi097.stdout: "user_id": "foo.client.0", 2023-01-24T16:38:37.682 INFO:teuthology.orchestra.run.smithi097.stdout: "display_name": "Mr. foo.client.0", 2023-01-24T16:38:37.682 INFO:teuthology.orchestra.run.smithi097.stdout: "email": "foo.client.0+test@test.test", 2023-01-24T16:38:37.682 INFO:teuthology.orchestra.run.smithi097.stdout: "suspended": 0, 2023-01-24T16:38:37.683 INFO:teuthology.orchestra.run.smithi097.stdout: "max_buckets": 1000, 2023-01-24T16:38:37.683 INFO:teuthology.orchestra.run.smithi097.stdout: "subusers": [], 2023-01-24T16:38:37.683 INFO:teuthology.orchestra.run.smithi097.stdout: "keys": [ 2023-01-24T16:38:37.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2023-01-24T16:38:37.683 INFO:teuthology.orchestra.run.smithi097.stdout: "user": "foo.client.0", 2023-01-24T16:38:37.683 INFO:teuthology.orchestra.run.smithi097.stdout: "access_key": "BNAVUMGDINJPSXUTEDAY", 2023-01-24T16:38:37.684 INFO:teuthology.orchestra.run.smithi097.stdout: "secret_key": "R66pXyd6bdS8/nlLqk3nvlGoVoVS3Vfg4JaBpajzSgm6vRF3gOwOgQ==" 2023-01-24T16:38:37.684 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:37.684 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2023-01-24T16:38:37.684 INFO:teuthology.orchestra.run.smithi097.stdout: "swift_keys": [], 2023-01-24T16:38:37.684 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": [ 2023-01-24T16:38:37.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2023-01-24T16:38:37.685 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "user-policy", 2023-01-24T16:38:37.685 INFO:teuthology.orchestra.run.smithi097.stdout: "perm": "*" 2023-01-24T16:38:37.685 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:37.685 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2023-01-24T16:38:37.686 INFO:teuthology.orchestra.run.smithi097.stdout: "op_mask": "read, write, delete", 2023-01-24T16:38:37.686 INFO:teuthology.orchestra.run.smithi097.stdout: "default_placement": "", 2023-01-24T16:38:37.686 INFO:teuthology.orchestra.run.smithi097.stdout: "default_storage_class": "", 2023-01-24T16:38:37.686 INFO:teuthology.orchestra.run.smithi097.stdout: "placement_tags": [], 2023-01-24T16:38:37.687 INFO:teuthology.orchestra.run.smithi097.stdout: "bucket_quota": { 2023-01-24T16:38:37.687 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": false, 2023-01-24T16:38:37.687 INFO:teuthology.orchestra.run.smithi097.stdout: "check_on_raw": false, 2023-01-24T16:38:37.688 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size": -1, 2023-01-24T16:38:37.688 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size_kb": 0, 2023-01-24T16:38:37.688 INFO:teuthology.orchestra.run.smithi097.stdout: "max_objects": -1 2023-01-24T16:38:37.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:37.689 INFO:teuthology.orchestra.run.smithi097.stdout: "user_quota": { 2023-01-24T16:38:37.689 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": false, 2023-01-24T16:38:37.689 INFO:teuthology.orchestra.run.smithi097.stdout: "check_on_raw": false, 2023-01-24T16:38:37.689 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size": -1, 2023-01-24T16:38:37.689 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size_kb": 0, 2023-01-24T16:38:37.690 INFO:teuthology.orchestra.run.smithi097.stdout: "max_objects": -1 2023-01-24T16:38:37.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:37.690 INFO:teuthology.orchestra.run.smithi097.stdout: "temp_url_keys": [], 2023-01-24T16:38:37.690 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "rgw", 2023-01-24T16:38:37.690 INFO:teuthology.orchestra.run.smithi097.stdout: "mfa_ids": [] 2023-01-24T16:38:37.691 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-01-24T16:38:37.691 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:37.692 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.677+0000 7f25cc38c680 20 remove_watcher() i=0 2023-01-24T16:38:37.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.677+0000 7f25cc38c680 2 removed watcher, disabling cache 2023-01-24T16:38:37.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.679+0000 7f25cc38c680 20 remove_watcher() i=1 2023-01-24T16:38:37.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.680+0000 7f25cc38c680 20 remove_watcher() i=2 2023-01-24T16:38:37.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.682+0000 7f25cc38c680 20 remove_watcher() i=3 2023-01-24T16:38:37.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.682+0000 7f25cc38c680 20 remove_watcher() i=4 2023-01-24T16:38:37.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.683+0000 7f25cc38c680 20 remove_watcher() i=5 2023-01-24T16:38:37.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.684+0000 7f25cc38c680 20 remove_watcher() i=6 2023-01-24T16:38:37.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.685+0000 7f25cc38c680 20 remove_watcher() i=7 2023-01-24T16:38:37.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.686+0000 7f250bfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:37.695 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 9776660651 --totp-seed CWUK3KHSU5WCC56RDE6FYOVBLKSRVHUVYDBQ4PV2E7G5TIRGU2GHOHDDDV3WMAMJ --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-01-24T16:38:37.742 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:38:37.742 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:38:37.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.746+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.747+0000 7faf46361680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.747+0000 7faf46361680 20 realm 2023-01-24T16:38:37.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.747+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.748+0000 7faf46361680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.748+0000 7faf46361680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T16:38:37.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.748+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.748+0000 7faf46361680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.748+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.748+0000 7faf46361680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:37.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.748+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.748+0000 7faf46361680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:38:37.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.748+0000 7faf46361680 20 searching for the correct realm 2023-01-24T16:38:37.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.755+0000 7faf46361680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:37.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.755+0000 7faf46361680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:37.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.755+0000 7faf46361680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:37.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.755+0000 7faf46361680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:37.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.755+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.756+0000 7faf46361680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.756+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.756+0000 7faf46361680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:37.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.756+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.756+0000 7faf46361680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:37.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.756+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.756+0000 7faf46361680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.756+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.756+0000 7faf46361680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.756+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.757+0000 7faf46361680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:37.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.763+0000 7faf46361680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:37.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.763+0000 7faf46361680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:37.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.763+0000 7faf46361680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:37.766 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.763+0000 7faf46361680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:37.766 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.763+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.766 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.763+0000 7faf46361680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:37.766 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.763+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.766 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.763+0000 7faf46361680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:38:37.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.763+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.763+0000 7faf46361680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:37.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.763+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:37.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 zone default found 2023-01-24T16:38:37.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 4 Realm: () 2023-01-24T16:38:37.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 4 ZoneGroup: default (2097a68d-863f-4181-8202-77da07845417) 2023-01-24T16:38:37.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 4 Zone: default (e9b1a570-bc22-4f42-b7c6-96dbb40f320d) 2023-01-24T16:38:37.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T16:38:37.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 zonegroup default 2023-01-24T16:38:37.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:37.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 started sync module instance, tier type = 2023-01-24T16:38:37.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.764+0000 7faf46361680 20 started zone id=e9b1a570-bc22-4f42-b7c6-96dbb40f320d (name=default) with tier type = 2023-01-24T16:38:37.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.781+0000 7faf46361680 20 add_watcher() i=0 2023-01-24T16:38:37.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.781+0000 7faf46361680 20 add_watcher() i=1 2023-01-24T16:38:37.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.781+0000 7faf46361680 20 add_watcher() i=2 2023-01-24T16:38:37.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.781+0000 7faf46361680 20 add_watcher() i=3 2023-01-24T16:38:37.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.781+0000 7faf46361680 20 add_watcher() i=4 2023-01-24T16:38:37.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.781+0000 7faf46361680 20 add_watcher() i=5 2023-01-24T16:38:37.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.781+0000 7faf46361680 20 add_watcher() i=6 2023-01-24T16:38:37.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.783+0000 7faf46361680 20 add_watcher() i=7 2023-01-24T16:38:37.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.783+0000 7faf46361680 2 all 8 watchers are set, enabling cache 2023-01-24T16:38:37.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.784+0000 7faf0a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:37.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.784+0000 7faf46361680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:38:37.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.784+0000 7faf46361680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:38:37.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.788+0000 7faf46361680 5 note: GC not initialized 2023-01-24T16:38:37.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.789+0000 7fae797fa700 20 reqs_thread_entry: start 2023-01-24T16:38:37.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.789+0000 7fae72ffd700 20 reqs_thread_entry: start 2023-01-24T16:38:37.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.789+0000 7faf46361680 20 init_complete bucket index max shards: 11 2023-01-24T16:38:37.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.789+0000 7faf46361680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T16:38:37.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.789+0000 7fae717fa700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 16:39:07 2023 2023-01-24T16:38:37.791 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:38:37.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.789+0000 7fae70ff9700 20 reqs_thread_entry: start 2023-01-24T16:38:37.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.789+0000 7faf46361680 20 get_system_obj_state: rctx=0x7ffdb1a72db8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55b1306e0940 s->prefetch_data=0 2023-01-24T16:38:37.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.789+0000 7faf46361680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-01-24T16:38:37.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.790+0000 7faf46361680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-01-24T16:38:37.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.790+0000 7faf46361680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T16:38:37.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.790+0000 7faf46361680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T16:38:37.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.790+0000 7faf46361680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T16:38:37.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.790+0000 7faf46361680 20 Read xattr: user.rgw.idtag 2023-01-24T16:38:37.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.790+0000 7faf46361680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T16:38:37.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.790+0000 7faf46361680 20 rados->read ofs=0 len=0 2023-01-24T16:38:37.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.790+0000 7faf46361680 20 rados_obj.operate() r=0 bl.length=455 2023-01-24T16:38:37.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.790+0000 7faf46361680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-01-24T16:38:37.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.790+0000 7faf46361680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T16:38:37.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:37.790+0000 7faf46361680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T16:38:39.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.362+0000 7faf46361680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T16:38:39.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.362+0000 7faf46361680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T16:38:39.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.362+0000 7faf46361680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-01-24T16:38:39.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.362+0000 7faf2a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 94219511686736 notifier 4245 bl.length()=654 2023-01-24T16:38:39.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.362+0000 7faf2a7fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T16:38:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.362+0000 7faf2a7fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T16:38:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.364+0000 7faf46361680 20 remove_watcher() i=0 2023-01-24T16:38:39.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.364+0000 7faf46361680 2 removed watcher, disabling cache 2023-01-24T16:38:39.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.365+0000 7faf46361680 20 remove_watcher() i=1 2023-01-24T16:38:39.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.367+0000 7faf46361680 20 remove_watcher() i=2 2023-01-24T16:38:39.370 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.370+0000 7faf46361680 20 remove_watcher() i=3 2023-01-24T16:38:39.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.373+0000 7faf46361680 20 remove_watcher() i=4 2023-01-24T16:38:39.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.374+0000 7faf46361680 20 remove_watcher() i=5 2023-01-24T16:38:39.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.376+0000 7faf46361680 20 remove_watcher() i=6 2023-01-24T16:38:39.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.378+0000 7faf46361680 20 remove_watcher() i=7 2023-01-24T16:38:39.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.379+0000 7faf0a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:39.385 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2023-01-24T16:38:39.386 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key HSDBKVCSJGJVYDFCJHPD --secret Mb6bXiGqqvAWWi0GXpiHBh+AM626/cIhS10EScgmDz7hBOXIJGzvZg== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-01-24T16:38:39.436 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:38:39.437 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:38:39.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.442+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.443+0000 7f6cabbbb680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.443+0000 7f6cabbbb680 20 realm 2023-01-24T16:38:39.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.443+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.443+0000 7f6cabbbb680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.443+0000 7f6cabbbb680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T16:38:39.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.443+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.443+0000 7f6cabbbb680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.443+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.443+0000 7f6cabbbb680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.443+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.444+0000 7f6cabbbb680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:38:39.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.444+0000 7f6cabbbb680 20 searching for the correct realm 2023-01-24T16:38:39.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.451+0000 7f6cabbbb680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:39.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.451+0000 7f6cabbbb680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:39.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.451+0000 7f6cabbbb680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:39.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.451+0000 7f6cabbbb680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:39.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.451+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.451+0000 7f6cabbbb680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.451+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.452+0000 7f6cabbbb680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.452+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.452+0000 7f6cabbbb680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:39.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.452+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.452+0000 7f6cabbbb680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.452+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.452+0000 7f6cabbbb680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.452+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.452+0000 7f6cabbbb680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:39.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:39.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:39.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:39.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:38:39.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.459+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 20 zone default found 2023-01-24T16:38:39.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 4 Realm: () 2023-01-24T16:38:39.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 4 ZoneGroup: default (2097a68d-863f-4181-8202-77da07845417) 2023-01-24T16:38:39.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 4 Zone: default (e9b1a570-bc22-4f42-b7c6-96dbb40f320d) 2023-01-24T16:38:39.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T16:38:39.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 20 zonegroup default 2023-01-24T16:38:39.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 20 started sync module instance, tier type = 2023-01-24T16:38:39.466 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.460+0000 7f6cabbbb680 20 started zone id=e9b1a570-bc22-4f42-b7c6-96dbb40f320d (name=default) with tier type = 2023-01-24T16:38:39.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.480+0000 7f6cabbbb680 20 add_watcher() i=0 2023-01-24T16:38:39.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.480+0000 7f6cabbbb680 20 add_watcher() i=1 2023-01-24T16:38:39.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.480+0000 7f6cabbbb680 20 add_watcher() i=2 2023-01-24T16:38:39.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.480+0000 7f6cabbbb680 20 add_watcher() i=3 2023-01-24T16:38:39.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.480+0000 7f6cabbbb680 20 add_watcher() i=4 2023-01-24T16:38:39.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.480+0000 7f6cabbbb680 20 add_watcher() i=5 2023-01-24T16:38:39.483 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.480+0000 7f6cabbbb680 20 add_watcher() i=6 2023-01-24T16:38:39.483 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.482+0000 7f6cabbbb680 20 add_watcher() i=7 2023-01-24T16:38:39.483 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.482+0000 7f6cabbbb680 2 all 8 watchers are set, enabling cache 2023-01-24T16:38:39.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.484+0000 7f6bebfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:39.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.484+0000 7f6cabbbb680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:38:39.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.484+0000 7f6cabbbb680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:38:39.488 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.487+0000 7f6cabbbb680 5 note: GC not initialized 2023-01-24T16:38:39.490 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.488+0000 7f6bdaffd700 20 reqs_thread_entry: start 2023-01-24T16:38:39.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.488+0000 7f6bd8ff9700 20 reqs_thread_entry: start 2023-01-24T16:38:39.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6cabbbb680 20 init_complete bucket index max shards: 11 2023-01-24T16:38:39.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6cabbbb680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T16:38:39.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6bd2ffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 16:39:09 2023 2023-01-24T16:38:39.491 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:38:39.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6bd27fc700 20 reqs_thread_entry: start 2023-01-24T16:38:39.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6cabbbb680 20 get_system_obj_state: rctx=0x7ffe4be21ce8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55723b2e1940 s->prefetch_data=0 2023-01-24T16:38:39.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6cabbbb680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-01-24T16:38:39.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6cabbbb680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2023-01-24T16:38:39.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6cabbbb680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T16:38:39.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6cabbbb680 20 get_system_obj_state: rctx=0x7ffe4be219d8 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x55723b2e1940 s->prefetch_data=0 2023-01-24T16:38:39.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6cabbbb680 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2023-01-24T16:38:39.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6cabbbb680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2023-01-24T16:38:39.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6cabbbb680 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-01-24T16:38:39.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6cabbbb680 20 get_system_obj_state: rctx=0x7ffe4be219d8 obj=default.rgw.meta:users.keys:HSDBKVCSJGJVYDFCJHPD state=0x55723b2e1940 s->prefetch_data=0 2023-01-24T16:38:39.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.489+0000 7f6cabbbb680 10 cache get: name=default.rgw.meta+users.keys+HSDBKVCSJGJVYDFCJHPD : miss 2023-01-24T16:38:39.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.490+0000 7f6cabbbb680 10 cache put: name=default.rgw.meta+users.keys+HSDBKVCSJGJVYDFCJHPD info.flags=0x0 2023-01-24T16:38:39.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.490+0000 7f6cabbbb680 10 adding default.rgw.meta+users.keys+HSDBKVCSJGJVYDFCJHPD to cache LRU end 2023-01-24T16:38:39.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.490+0000 7f6cabbbb680 20 get_system_obj_state: rctx=0x7ffe4be21268 obj=default.rgw.meta:users.keys:HSDBKVCSJGJVYDFCJHPD state=0x55723b2e1940 s->prefetch_data=0 2023-01-24T16:38:39.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.490+0000 7f6cabbbb680 10 cache get: name=default.rgw.meta+users.keys+HSDBKVCSJGJVYDFCJHPD : hit (negative entry) 2023-01-24T16:38:39.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.490+0000 7f6cabbbb680 20 get_system_obj_state: rctx=0x7ffe4be21918 obj=default.rgw.meta:users.keys:HSDBKVCSJGJVYDFCJHPD state=0x55723b2e1940 s->prefetch_data=0 2023-01-24T16:38:39.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.490+0000 7f6cabbbb680 10 cache get: name=default.rgw.meta+users.keys+HSDBKVCSJGJVYDFCJHPD : hit (negative entry) 2023-01-24T16:38:39.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.491+0000 7f6cabbbb680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T16:38:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.491+0000 7f6cabbbb680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T16:38:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.491+0000 7f6cabbbb680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-01-24T16:38:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.491+0000 7f6c9ccb9700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116993 cookie 93949109109968 notifier 4251 bl.length()=640 2023-01-24T16:38:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.491+0000 7f6c9ccb9700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T16:38:39.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.491+0000 7f6c9ccb9700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T16:38:39.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.492+0000 7f6cabbbb680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-01-24T16:38:39.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.492+0000 7f6cabbbb680 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-01-24T16:38:39.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.492+0000 7f6cabbbb680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-01-24T16:38:39.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.493+0000 7f6c97fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 93949109109968 notifier 4251 bl.length()=194 2023-01-24T16:38:39.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.493+0000 7f6c97fff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-01-24T16:38:39.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.493+0000 7f6c97fff700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-01-24T16:38:39.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.494+0000 7f6cabbbb680 10 cache put: name=default.rgw.meta+users.keys+HSDBKVCSJGJVYDFCJHPD info.flags=0x7 2023-01-24T16:38:39.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.494+0000 7f6cabbbb680 10 moving default.rgw.meta+users.keys+HSDBKVCSJGJVYDFCJHPD to cache LRU end 2023-01-24T16:38:39.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.494+0000 7f6cabbbb680 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.keys:HSDBKVCSJGJVYDFCJHPD, ofs0, ns] 2023-01-24T16:38:39.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.494+0000 7f6c97fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116992 cookie 93949109107760 notifier 4251 bl.length()=186 2023-01-24T16:38:39.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.494+0000 7f6c97fff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+HSDBKVCSJGJVYDFCJHPD info.flags=0x7 2023-01-24T16:38:39.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.494+0000 7f6c97fff700 10 rgw watcher librados: moving default.rgw.meta+users.keys+HSDBKVCSJGJVYDFCJHPD to cache LRU end 2023-01-24T16:38:39.501 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-01-24T16:38:39.501 INFO:teuthology.orchestra.run.smithi097.stdout: "user_id": "bar.client.0", 2023-01-24T16:38:39.501 INFO:teuthology.orchestra.run.smithi097.stdout: "display_name": "Mr. bar.client.0", 2023-01-24T16:38:39.502 INFO:teuthology.orchestra.run.smithi097.stdout: "email": "bar.client.0+test@test.test", 2023-01-24T16:38:39.502 INFO:teuthology.orchestra.run.smithi097.stdout: "suspended": 0, 2023-01-24T16:38:39.502 INFO:teuthology.orchestra.run.smithi097.stdout: "max_buckets": 1000, 2023-01-24T16:38:39.502 INFO:teuthology.orchestra.run.smithi097.stdout: "subusers": [], 2023-01-24T16:38:39.502 INFO:teuthology.orchestra.run.smithi097.stdout: "keys": [ 2023-01-24T16:38:39.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2023-01-24T16:38:39.503 INFO:teuthology.orchestra.run.smithi097.stdout: "user": "bar.client.0", 2023-01-24T16:38:39.503 INFO:teuthology.orchestra.run.smithi097.stdout: "access_key": "HSDBKVCSJGJVYDFCJHPD", 2023-01-24T16:38:39.503 INFO:teuthology.orchestra.run.smithi097.stdout: "secret_key": "Mb6bXiGqqvAWWi0GXpiHBh+AM626/cIhS10EScgmDz7hBOXIJGzvZg==" 2023-01-24T16:38:39.503 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:39.504 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2023-01-24T16:38:39.504 INFO:teuthology.orchestra.run.smithi097.stdout: "swift_keys": [], 2023-01-24T16:38:39.504 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": [ 2023-01-24T16:38:39.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2023-01-24T16:38:39.504 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "user-policy", 2023-01-24T16:38:39.505 INFO:teuthology.orchestra.run.smithi097.stdout: "perm": "*" 2023-01-24T16:38:39.505 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:39.505 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2023-01-24T16:38:39.505 INFO:teuthology.orchestra.run.smithi097.stdout: "op_mask": "read, write, delete", 2023-01-24T16:38:39.505 INFO:teuthology.orchestra.run.smithi097.stdout: "default_placement": "", 2023-01-24T16:38:39.506 INFO:teuthology.orchestra.run.smithi097.stdout: "default_storage_class": "", 2023-01-24T16:38:39.506 INFO:teuthology.orchestra.run.smithi097.stdout: "placement_tags": [], 2023-01-24T16:38:39.506 INFO:teuthology.orchestra.run.smithi097.stdout: "bucket_quota": { 2023-01-24T16:38:39.506 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": false, 2023-01-24T16:38:39.506 INFO:teuthology.orchestra.run.smithi097.stdout: "check_on_raw": false, 2023-01-24T16:38:39.507 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size": -1, 2023-01-24T16:38:39.507 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size_kb": 0, 2023-01-24T16:38:39.507 INFO:teuthology.orchestra.run.smithi097.stdout: "max_objects": -1 2023-01-24T16:38:39.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:39.507 INFO:teuthology.orchestra.run.smithi097.stdout: "user_quota": { 2023-01-24T16:38:39.507 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": false, 2023-01-24T16:38:39.508 INFO:teuthology.orchestra.run.smithi097.stdout: "check_on_raw": false, 2023-01-24T16:38:39.508 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size": -1, 2023-01-24T16:38:39.508 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size_kb": 0, 2023-01-24T16:38:39.508 INFO:teuthology.orchestra.run.smithi097.stdout: "max_objects": -1 2023-01-24T16:38:39.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:39.509 INFO:teuthology.orchestra.run.smithi097.stdout: "temp_url_keys": [], 2023-01-24T16:38:39.509 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "rgw", 2023-01-24T16:38:39.509 INFO:teuthology.orchestra.run.smithi097.stdout: "mfa_ids": [] 2023-01-24T16:38:39.510 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-01-24T16:38:39.510 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:39.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.496+0000 7f6cabbbb680 20 remove_watcher() i=0 2023-01-24T16:38:39.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.496+0000 7f6cabbbb680 2 removed watcher, disabling cache 2023-01-24T16:38:39.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.497+0000 7f6cabbbb680 20 remove_watcher() i=1 2023-01-24T16:38:39.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.497+0000 7f6cabbbb680 20 remove_watcher() i=2 2023-01-24T16:38:39.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.498+0000 7f6cabbbb680 20 remove_watcher() i=3 2023-01-24T16:38:39.513 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.499+0000 7f6cabbbb680 20 remove_watcher() i=4 2023-01-24T16:38:39.513 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.500+0000 7f6cabbbb680 20 remove_watcher() i=5 2023-01-24T16:38:39.513 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.501+0000 7f6cabbbb680 20 remove_watcher() i=6 2023-01-24T16:38:39.513 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.502+0000 7f6cabbbb680 20 remove_watcher() i=7 2023-01-24T16:38:39.513 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.502+0000 7f6bebfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:39.514 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 4806492086 --totp-seed BLGAEE7UMWKJ5N5ON6GLWBCZ55OXUPJKXYXUJUFBRM4GPD2CQJAGCHWXGXYQUBIW --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-01-24T16:38:39.561 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:38:39.562 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:38:39.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.566+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.567+0000 7fc10d3d5680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.567+0000 7fc10d3d5680 20 realm 2023-01-24T16:38:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.567+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.567+0000 7fc10d3d5680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.567+0000 7fc10d3d5680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T16:38:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.567+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.567+0000 7fc10d3d5680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.567+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.568+0000 7fc10d3d5680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.568+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.568+0000 7fc10d3d5680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:38:39.571 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.568+0000 7fc10d3d5680 20 searching for the correct realm 2023-01-24T16:38:39.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.575+0000 7fc10d3d5680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:39.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.575+0000 7fc10d3d5680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:39.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.575+0000 7fc10d3d5680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:39.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.575+0000 7fc10d3d5680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:39.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.575+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.578 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.575+0000 7fc10d3d5680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.578 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.575+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.578 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.575+0000 7fc10d3d5680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.578 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.575+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.578 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.576+0000 7fc10d3d5680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:39.578 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.576+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.576+0000 7fc10d3d5680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.576+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.576+0000 7fc10d3d5680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.576+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.576+0000 7fc10d3d5680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.582+0000 7fc10d3d5680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:39.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.582+0000 7fc10d3d5680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:39.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.582+0000 7fc10d3d5680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:39.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.582+0000 7fc10d3d5680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:39.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.582+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.582+0000 7fc10d3d5680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.582+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:38:39.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:39.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 20 zone default found 2023-01-24T16:38:39.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 4 Realm: () 2023-01-24T16:38:39.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 4 ZoneGroup: default (2097a68d-863f-4181-8202-77da07845417) 2023-01-24T16:38:39.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 4 Zone: default (e9b1a570-bc22-4f42-b7c6-96dbb40f320d) 2023-01-24T16:38:39.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T16:38:39.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 20 zonegroup default 2023-01-24T16:38:39.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.583+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.584+0000 7fc10d3d5680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.584+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.584+0000 7fc10d3d5680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.584+0000 7fc10d3d5680 20 started sync module instance, tier type = 2023-01-24T16:38:39.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.584+0000 7fc10d3d5680 20 started zone id=e9b1a570-bc22-4f42-b7c6-96dbb40f320d (name=default) with tier type = 2023-01-24T16:38:39.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.593+0000 7fc10d3d5680 20 add_watcher() i=0 2023-01-24T16:38:39.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.593+0000 7fc10d3d5680 20 add_watcher() i=1 2023-01-24T16:38:39.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.593+0000 7fc10d3d5680 20 add_watcher() i=2 2023-01-24T16:38:39.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.593+0000 7fc10d3d5680 20 add_watcher() i=3 2023-01-24T16:38:39.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.593+0000 7fc10d3d5680 20 add_watcher() i=4 2023-01-24T16:38:39.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.593+0000 7fc10d3d5680 20 add_watcher() i=5 2023-01-24T16:38:39.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.593+0000 7fc10d3d5680 20 add_watcher() i=6 2023-01-24T16:38:39.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.594+0000 7fc10d3d5680 20 add_watcher() i=7 2023-01-24T16:38:39.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.594+0000 7fc10d3d5680 2 all 8 watchers are set, enabling cache 2023-01-24T16:38:39.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.595+0000 7fc0d1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:39.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.595+0000 7fc10d3d5680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:38:39.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.595+0000 7fc10d3d5680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:38:39.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.599+0000 7fc10d3d5680 5 note: GC not initialized 2023-01-24T16:38:39.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.600+0000 7fc048ff9700 20 reqs_thread_entry: start 2023-01-24T16:38:39.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.600+0000 7fc03e7fc700 20 reqs_thread_entry: start 2023-01-24T16:38:39.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.600+0000 7fc10d3d5680 20 init_complete bucket index max shards: 11 2023-01-24T16:38:39.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.600+0000 7fc10d3d5680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T16:38:39.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.600+0000 7fc03cff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 16:39:10 2023 2023-01-24T16:38:39.603 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:38:39.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.600+0000 7fc023fff700 20 reqs_thread_entry: start 2023-01-24T16:38:39.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.600+0000 7fc10d3d5680 20 get_system_obj_state: rctx=0x7ffc517dba48 obj=default.rgw.meta:users.uid:bar.client.0 state=0x561c7adae940 s->prefetch_data=0 2023-01-24T16:38:39.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.600+0000 7fc10d3d5680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-01-24T16:38:39.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.601+0000 7fc10d3d5680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-01-24T16:38:39.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.601+0000 7fc10d3d5680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T16:38:39.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.601+0000 7fc10d3d5680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T16:38:39.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.601+0000 7fc10d3d5680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T16:38:39.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.601+0000 7fc10d3d5680 20 Read xattr: user.rgw.idtag 2023-01-24T16:38:39.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.601+0000 7fc10d3d5680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T16:38:39.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.601+0000 7fc10d3d5680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.601+0000 7fc10d3d5680 20 rados_obj.operate() r=0 bl.length=455 2023-01-24T16:38:39.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.601+0000 7fc10d3d5680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-01-24T16:38:39.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.601+0000 7fc10d3d5680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T16:38:39.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.601+0000 7fc10d3d5680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T16:38:39.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.603+0000 7fc10d3d5680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T16:38:39.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.603+0000 7fc10d3d5680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T16:38:39.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.603+0000 7fc10d3d5680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-01-24T16:38:39.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.604+0000 7fc0fe4d3700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 94680321844976 notifier 4257 bl.length()=654 2023-01-24T16:38:39.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.604+0000 7fc0fe4d3700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T16:38:39.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.604+0000 7fc0fe4d3700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T16:38:39.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.605+0000 7fc10d3d5680 20 remove_watcher() i=0 2023-01-24T16:38:39.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.605+0000 7fc10d3d5680 2 removed watcher, disabling cache 2023-01-24T16:38:39.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.606+0000 7fc10d3d5680 20 remove_watcher() i=1 2023-01-24T16:38:39.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.607+0000 7fc10d3d5680 20 remove_watcher() i=2 2023-01-24T16:38:39.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.607+0000 7fc10d3d5680 20 remove_watcher() i=3 2023-01-24T16:38:39.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.610+0000 7fc10d3d5680 20 remove_watcher() i=4 2023-01-24T16:38:39.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.611+0000 7fc10d3d5680 20 remove_watcher() i=5 2023-01-24T16:38:39.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.613+0000 7fc10d3d5680 20 remove_watcher() i=6 2023-01-24T16:38:39.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.616+0000 7fc10d3d5680 20 remove_watcher() i=7 2023-01-24T16:38:39.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.617+0000 7fc0d1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:39.623 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2023-01-24T16:38:39.623 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key TFMNMIQKIHDFPLQXWVRM --secret QVczMXWorJxteqYCfl1561CjrDFMOe4yDiSIba8bl0gaYl0UnAFGqg== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2023-01-24T16:38:39.671 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:38:39.671 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:38:39.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.675+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.676+0000 7fa86585d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.676+0000 7fa86585d680 20 realm 2023-01-24T16:38:39.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.676+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.677+0000 7fa86585d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.677+0000 7fa86585d680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T16:38:39.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.677+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.677+0000 7fa86585d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.677+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.677+0000 7fa86585d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.680 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.677+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.680 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.677+0000 7fa86585d680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:38:39.680 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.677+0000 7fa86585d680 20 searching for the correct realm 2023-01-24T16:38:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.684+0000 7fa86585d680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.684+0000 7fa86585d680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.684+0000 7fa86585d680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:39.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.684+0000 7fa86585d680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:39.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.684+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.685+0000 7fa86585d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.685+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.685+0000 7fa86585d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.685+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.685+0000 7fa86585d680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.685+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.685+0000 7fa86585d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.685+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.685+0000 7fa86585d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.685+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.686+0000 7fa86585d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.691+0000 7fa86585d680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:39.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.691+0000 7fa86585d680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:39.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.691+0000 7fa86585d680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:39.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.691+0000 7fa86585d680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:39.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.691+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.692+0000 7fa86585d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.692+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.692+0000 7fa86585d680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:38:39.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.692+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.692+0000 7fa86585d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.692+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.692+0000 7fa86585d680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:39.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.692+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.697 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.697 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 20 zone default found 2023-01-24T16:38:39.697 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 4 Realm: () 2023-01-24T16:38:39.697 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 4 ZoneGroup: default (2097a68d-863f-4181-8202-77da07845417) 2023-01-24T16:38:39.697 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 4 Zone: default (e9b1a570-bc22-4f42-b7c6-96dbb40f320d) 2023-01-24T16:38:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T16:38:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 20 zonegroup default 2023-01-24T16:38:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 20 started sync module instance, tier type = 2023-01-24T16:38:39.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.693+0000 7fa86585d680 20 started zone id=e9b1a570-bc22-4f42-b7c6-96dbb40f320d (name=default) with tier type = 2023-01-24T16:38:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.722+0000 7fa86585d680 20 add_watcher() i=0 2023-01-24T16:38:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.722+0000 7fa86585d680 20 add_watcher() i=1 2023-01-24T16:38:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.722+0000 7fa86585d680 20 add_watcher() i=2 2023-01-24T16:38:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.722+0000 7fa86585d680 20 add_watcher() i=3 2023-01-24T16:38:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.722+0000 7fa86585d680 20 add_watcher() i=4 2023-01-24T16:38:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.722+0000 7fa86585d680 20 add_watcher() i=5 2023-01-24T16:38:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.722+0000 7fa86585d680 20 add_watcher() i=6 2023-01-24T16:38:39.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.726+0000 7fa86585d680 20 add_watcher() i=7 2023-01-24T16:38:39.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.726+0000 7fa86585d680 2 all 8 watchers are set, enabling cache 2023-01-24T16:38:39.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.730+0000 7fa829fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:39.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.730+0000 7fa86585d680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:38:39.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.730+0000 7fa86585d680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:38:39.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.733+0000 7fa86585d680 5 note: GC not initialized 2023-01-24T16:38:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.734+0000 7fa7a0ff9700 20 reqs_thread_entry: start 2023-01-24T16:38:39.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa7967fc700 20 reqs_thread_entry: start 2023-01-24T16:38:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa86585d680 20 init_complete bucket index max shards: 11 2023-01-24T16:38:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa86585d680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T16:38:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa794ff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 16:39:10 2023 2023-01-24T16:38:39.737 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:38:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa77bfff700 20 reqs_thread_entry: start 2023-01-24T16:38:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa86585d680 20 get_system_obj_state: rctx=0x7fff181337e8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55d60d9ab940 s->prefetch_data=0 2023-01-24T16:38:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa86585d680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-01-24T16:38:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa86585d680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2023-01-24T16:38:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa86585d680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T16:38:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa86585d680 20 get_system_obj_state: rctx=0x7fff181334d8 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x55d60d9ab940 s->prefetch_data=0 2023-01-24T16:38:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa86585d680 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2023-01-24T16:38:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa86585d680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2023-01-24T16:38:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa86585d680 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-01-24T16:38:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa86585d680 20 get_system_obj_state: rctx=0x7fff181334d8 obj=default.rgw.meta:users.keys:TFMNMIQKIHDFPLQXWVRM state=0x55d60d9ab940 s->prefetch_data=0 2023-01-24T16:38:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.735+0000 7fa86585d680 10 cache get: name=default.rgw.meta+users.keys+TFMNMIQKIHDFPLQXWVRM : miss 2023-01-24T16:38:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.736+0000 7fa86585d680 10 cache put: name=default.rgw.meta+users.keys+TFMNMIQKIHDFPLQXWVRM info.flags=0x0 2023-01-24T16:38:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.736+0000 7fa86585d680 10 adding default.rgw.meta+users.keys+TFMNMIQKIHDFPLQXWVRM to cache LRU end 2023-01-24T16:38:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.736+0000 7fa86585d680 20 get_system_obj_state: rctx=0x7fff18132d68 obj=default.rgw.meta:users.keys:TFMNMIQKIHDFPLQXWVRM state=0x55d60d9ab940 s->prefetch_data=0 2023-01-24T16:38:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.736+0000 7fa86585d680 10 cache get: name=default.rgw.meta+users.keys+TFMNMIQKIHDFPLQXWVRM : hit (negative entry) 2023-01-24T16:38:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.736+0000 7fa86585d680 20 get_system_obj_state: rctx=0x7fff18133418 obj=default.rgw.meta:users.keys:TFMNMIQKIHDFPLQXWVRM state=0x55d60d9ab940 s->prefetch_data=0 2023-01-24T16:38:39.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.736+0000 7fa86585d680 10 cache get: name=default.rgw.meta+users.keys+TFMNMIQKIHDFPLQXWVRM : hit (negative entry) 2023-01-24T16:38:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.737+0000 7fa86585d680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T16:38:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.737+0000 7fa86585d680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T16:38:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.737+0000 7fa86585d680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-01-24T16:38:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.737+0000 7fa8527fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 94377841225408 notifier 4263 bl.length()=714 2023-01-24T16:38:39.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.737+0000 7fa8527fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T16:38:39.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.737+0000 7fa8527fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T16:38:39.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.741+0000 7fa86585d680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-01-24T16:38:39.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.741+0000 7fa86585d680 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-01-24T16:38:39.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.741+0000 7fa86585d680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-01-24T16:38:39.744 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.741+0000 7fa8527fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94377841230352 notifier 4263 bl.length()=224 2023-01-24T16:38:39.744 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.741+0000 7fa8527fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-01-24T16:38:39.744 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.741+0000 7fa8527fc700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-01-24T16:38:39.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.742+0000 7fa86585d680 10 cache put: name=default.rgw.meta+users.keys+TFMNMIQKIHDFPLQXWVRM info.flags=0x7 2023-01-24T16:38:39.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.742+0000 7fa86585d680 10 moving default.rgw.meta+users.keys+TFMNMIQKIHDFPLQXWVRM to cache LRU end 2023-01-24T16:38:39.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.742+0000 7fa86585d680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:TFMNMIQKIHDFPLQXWVRM, ofs0, ns] 2023-01-24T16:38:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.742+0000 7fa8527fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94377841210080 notifier 4263 bl.length()=201 2023-01-24T16:38:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.742+0000 7fa8527fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+TFMNMIQKIHDFPLQXWVRM info.flags=0x7 2023-01-24T16:38:39.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.742+0000 7fa8527fc700 10 rgw watcher librados: moving default.rgw.meta+users.keys+TFMNMIQKIHDFPLQXWVRM to cache LRU end 2023-01-24T16:38:39.746 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-01-24T16:38:39.747 INFO:teuthology.orchestra.run.smithi097.stdout: "user_id": "testx$tenanteduser.client.0", 2023-01-24T16:38:39.747 INFO:teuthology.orchestra.run.smithi097.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2023-01-24T16:38:39.747 INFO:teuthology.orchestra.run.smithi097.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2023-01-24T16:38:39.747 INFO:teuthology.orchestra.run.smithi097.stdout: "suspended": 0, 2023-01-24T16:38:39.748 INFO:teuthology.orchestra.run.smithi097.stdout: "max_buckets": 1000, 2023-01-24T16:38:39.748 INFO:teuthology.orchestra.run.smithi097.stdout: "subusers": [], 2023-01-24T16:38:39.748 INFO:teuthology.orchestra.run.smithi097.stdout: "keys": [ 2023-01-24T16:38:39.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2023-01-24T16:38:39.748 INFO:teuthology.orchestra.run.smithi097.stdout: "user": "testx$tenanteduser.client.0", 2023-01-24T16:38:39.749 INFO:teuthology.orchestra.run.smithi097.stdout: "access_key": "TFMNMIQKIHDFPLQXWVRM", 2023-01-24T16:38:39.749 INFO:teuthology.orchestra.run.smithi097.stdout: "secret_key": "QVczMXWorJxteqYCfl1561CjrDFMOe4yDiSIba8bl0gaYl0UnAFGqg==" 2023-01-24T16:38:39.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:39.749 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2023-01-24T16:38:39.749 INFO:teuthology.orchestra.run.smithi097.stdout: "swift_keys": [], 2023-01-24T16:38:39.750 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": [ 2023-01-24T16:38:39.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2023-01-24T16:38:39.750 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "user-policy", 2023-01-24T16:38:39.750 INFO:teuthology.orchestra.run.smithi097.stdout: "perm": "*" 2023-01-24T16:38:39.750 INFO:teuthology.orchestra.run.smithi097.stdout: } 2023-01-24T16:38:39.751 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2023-01-24T16:38:39.751 INFO:teuthology.orchestra.run.smithi097.stdout: "op_mask": "read, write, delete", 2023-01-24T16:38:39.751 INFO:teuthology.orchestra.run.smithi097.stdout: "default_placement": "", 2023-01-24T16:38:39.751 INFO:teuthology.orchestra.run.smithi097.stdout: "default_storage_class": "", 2023-01-24T16:38:39.751 INFO:teuthology.orchestra.run.smithi097.stdout: "placement_tags": [], 2023-01-24T16:38:39.752 INFO:teuthology.orchestra.run.smithi097.stdout: "bucket_quota": { 2023-01-24T16:38:39.752 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": false, 2023-01-24T16:38:39.752 INFO:teuthology.orchestra.run.smithi097.stdout: "check_on_raw": false, 2023-01-24T16:38:39.752 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size": -1, 2023-01-24T16:38:39.752 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size_kb": 0, 2023-01-24T16:38:39.753 INFO:teuthology.orchestra.run.smithi097.stdout: "max_objects": -1 2023-01-24T16:38:39.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:39.753 INFO:teuthology.orchestra.run.smithi097.stdout: "user_quota": { 2023-01-24T16:38:39.753 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": false, 2023-01-24T16:38:39.753 INFO:teuthology.orchestra.run.smithi097.stdout: "check_on_raw": false, 2023-01-24T16:38:39.754 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size": -1, 2023-01-24T16:38:39.754 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size_kb": 0, 2023-01-24T16:38:39.754 INFO:teuthology.orchestra.run.smithi097.stdout: "max_objects": -1 2023-01-24T16:38:39.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2023-01-24T16:38:39.754 INFO:teuthology.orchestra.run.smithi097.stdout: "temp_url_keys": [], 2023-01-24T16:38:39.755 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "rgw", 2023-01-24T16:38:39.755 INFO:teuthology.orchestra.run.smithi097.stdout: "mfa_ids": [] 2023-01-24T16:38:39.755 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-01-24T16:38:39.755 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:39.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.744+0000 7fa86585d680 20 remove_watcher() i=0 2023-01-24T16:38:39.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.744+0000 7fa86585d680 2 removed watcher, disabling cache 2023-01-24T16:38:39.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.745+0000 7fa86585d680 20 remove_watcher() i=1 2023-01-24T16:38:39.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.746+0000 7fa86585d680 20 remove_watcher() i=2 2023-01-24T16:38:39.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.747+0000 7fa86585d680 20 remove_watcher() i=3 2023-01-24T16:38:39.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.748+0000 7fa86585d680 20 remove_watcher() i=4 2023-01-24T16:38:39.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.752+0000 7fa86585d680 20 remove_watcher() i=5 2023-01-24T16:38:39.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.756+0000 7fa86585d680 20 remove_watcher() i=6 2023-01-24T16:38:39.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.757+0000 7fa86585d680 20 remove_watcher() i=7 2023-01-24T16:38:39.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.758+0000 7fa829fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:39.764 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 6106177797 --totp-seed YVL6NIBGCSQTX34SVD3JJU6XYROEW6X5DCML7IMUQRFSX5GSLYFUUJPAZ6RNO7CA --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-01-24T16:38:39.811 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:38:39.811 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:38:39.816 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.815+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.818 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.817+0000 7f59cff5c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.818 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.817+0000 7f59cff5c680 20 realm 2023-01-24T16:38:39.818 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.817+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.818 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.817+0000 7f59cff5c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.818 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.817+0000 7f59cff5c680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T16:38:39.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.817+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.817+0000 7f59cff5c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.817+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.817+0000 7f59cff5c680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.817+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.820 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.817+0000 7f59cff5c680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:38:39.820 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.817+0000 7f59cff5c680 20 searching for the correct realm 2023-01-24T16:38:39.826 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.824+0000 7f59cff5c680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:39.826 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.824+0000 7f59cff5c680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:39.826 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.824+0000 7f59cff5c680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:39.827 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.824+0000 7f59cff5c680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:39.827 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.824+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.827 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.825+0000 7f59cff5c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.827 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.825+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.827 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.825+0000 7f59cff5c680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.828 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.825+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.828 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.825+0000 7f59cff5c680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:39.828 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.825+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.828 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.825+0000 7f59cff5c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.828 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.825+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.825+0000 7f59cff5c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.825+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.826+0000 7f59cff5c680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.831+0000 7f59cff5c680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:38:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.831+0000 7f59cff5c680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:38:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.831+0000 7f59cff5c680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:38:39.834 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.831+0000 7f59cff5c680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:38:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.831+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:38:39.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:38:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:38:39.836 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.837 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.837 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 20 zone default found 2023-01-24T16:38:39.837 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 4 Realm: () 2023-01-24T16:38:39.837 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 4 ZoneGroup: default (2097a68d-863f-4181-8202-77da07845417) 2023-01-24T16:38:39.837 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 4 Zone: default (e9b1a570-bc22-4f42-b7c6-96dbb40f320d) 2023-01-24T16:38:39.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T16:38:39.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 20 zonegroup default 2023-01-24T16:38:39.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.832+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.833+0000 7f59cff5c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.833+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.833+0000 7f59cff5c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.833+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.833+0000 7f59cff5c680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:38:39.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.833+0000 7f59cff5c680 20 started sync module instance, tier type = 2023-01-24T16:38:39.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.833+0000 7f59cff5c680 20 started zone id=e9b1a570-bc22-4f42-b7c6-96dbb40f320d (name=default) with tier type = 2023-01-24T16:38:39.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.860+0000 7f59cff5c680 20 add_watcher() i=0 2023-01-24T16:38:39.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.860+0000 7f59cff5c680 20 add_watcher() i=1 2023-01-24T16:38:39.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.860+0000 7f59cff5c680 20 add_watcher() i=2 2023-01-24T16:38:39.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.860+0000 7f59cff5c680 20 add_watcher() i=3 2023-01-24T16:38:39.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.860+0000 7f59cff5c680 20 add_watcher() i=4 2023-01-24T16:38:39.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.860+0000 7f59cff5c680 20 add_watcher() i=5 2023-01-24T16:38:39.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.860+0000 7f59cff5c680 20 add_watcher() i=6 2023-01-24T16:38:39.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.861+0000 7f59cff5c680 20 add_watcher() i=7 2023-01-24T16:38:39.864 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.861+0000 7f59cff5c680 2 all 8 watchers are set, enabling cache 2023-01-24T16:38:39.864 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.862+0000 7f5994fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:39.864 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.863+0000 7f59cff5c680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:38:39.864 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.863+0000 7f59cff5c680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:38:39.867 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.866+0000 7f59cff5c680 5 note: GC not initialized 2023-01-24T16:38:39.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.867+0000 7f59037fe700 20 reqs_thread_entry: start 2023-01-24T16:38:39.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.867+0000 7f59017fa700 20 reqs_thread_entry: start 2023-01-24T16:38:39.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.867+0000 7f59cff5c680 20 init_complete bucket index max shards: 11 2023-01-24T16:38:39.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.867+0000 7f59cff5c680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T16:38:39.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.867+0000 7f58ef7fe700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 16:39:10 2023 2023-01-24T16:38:39.870 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:38:39.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f58eeffd700 20 reqs_thread_entry: start 2023-01-24T16:38:39.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 20 get_system_obj_state: rctx=0x7ffffb4b3a58 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55bc13933940 s->prefetch_data=0 2023-01-24T16:38:39.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-01-24T16:38:39.871 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-01-24T16:38:39.871 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T16:38:39.871 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T16:38:39.871 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T16:38:39.872 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 20 Read xattr: user.rgw.idtag 2023-01-24T16:38:39.872 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T16:38:39.872 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 20 rados->read ofs=0 len=0 2023-01-24T16:38:39.873 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 20 rados_obj.operate() r=0 bl.length=514 2023-01-24T16:38:39.873 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-01-24T16:38:39.873 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T16:38:39.874 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.868+0000 7f59cff5c680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T16:38:39.874 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.871+0000 7f59cff5c680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T16:38:39.874 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.871+0000 7f59cff5c680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T16:38:39.874 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.871+0000 7f59cff5c680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-01-24T16:38:39.875 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.871+0000 7f59c185b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116997 cookie 94266272250112 notifier 4269 bl.length()=728 2023-01-24T16:38:39.875 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.871+0000 7f59c185b700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T16:38:39.875 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.871+0000 7f59c185b700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T16:38:39.875 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.872+0000 7f59cff5c680 20 remove_watcher() i=0 2023-01-24T16:38:39.875 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.872+0000 7f59cff5c680 2 removed watcher, disabling cache 2023-01-24T16:38:39.876 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.874+0000 7f59cff5c680 20 remove_watcher() i=1 2023-01-24T16:38:39.876 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.875+0000 7f59cff5c680 20 remove_watcher() i=2 2023-01-24T16:38:39.877 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.877+0000 7f59cff5c680 20 remove_watcher() i=3 2023-01-24T16:38:39.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.878+0000 7f59cff5c680 20 remove_watcher() i=4 2023-01-24T16:38:39.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.879+0000 7f59cff5c680 20 remove_watcher() i=5 2023-01-24T16:38:39.881 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.881+0000 7f59cff5c680 20 remove_watcher() i=6 2023-01-24T16:38:39.882 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.882+0000 7f59cff5c680 20 remove_watcher() i=7 2023-01-24T16:38:39.883 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:38:39.882+0000 7f5994fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:38:39.887 INFO:tasks.s3tests:Configuring s3-tests... 2023-01-24T16:38:39.888 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest/s3-tests && ./bootstrap 2023-01-24T16:38:46.718 INFO:teuthology.orchestra.run.smithi097.stderr:./bootstrap: Missing required RPM packages: libevent-devel libffi-devel libxml2-devel libxslt-devel zlib-devel. 2023-01-24T16:38:47.071 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-01-24T16:38:47.904 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:38 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:38:49.677 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-01-24T16:38:49.679 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:38:49.679 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-01-24T16:38:49.679 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:38:49.679 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-01-24T16:38:49.680 INFO:teuthology.orchestra.run.smithi097.stdout: libevent-devel x86_64 2.1.8-5.el8 rhel-8-for-x86_64-appstream-rpms 104 k 2023-01-24T16:38:49.680 INFO:teuthology.orchestra.run.smithi097.stdout: libffi-devel x86_64 3.1-22.el8 rhel-8-for-x86_64-baseos-rpms 29 k 2023-01-24T16:38:49.680 INFO:teuthology.orchestra.run.smithi097.stdout: libxml2-devel x86_64 2.9.7-9.el8_4.2 rhel-8-for-x86_64-appstream-rpms 1.0 M 2023-01-24T16:38:49.680 INFO:teuthology.orchestra.run.smithi097.stdout: libxslt-devel x86_64 1.1.32-6.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-01-24T16:38:49.681 INFO:teuthology.orchestra.run.smithi097.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-01-24T16:38:49.681 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-01-24T16:38:49.681 INFO:teuthology.orchestra.run.smithi097.stdout: cmake-filesystem x86_64 3.18.2-11.el8_4 rhel-8-for-x86_64-appstream-rpms 44 k 2023-01-24T16:38:49.681 INFO:teuthology.orchestra.run.smithi097.stdout: libgcrypt-devel x86_64 1.8.5-4.el8 rhel-8-for-x86_64-baseos-rpms 149 k 2023-01-24T16:38:49.681 INFO:teuthology.orchestra.run.smithi097.stdout: libgpg-error-devel 2023-01-24T16:38:49.682 INFO:teuthology.orchestra.run.smithi097.stdout: x86_64 1.31-1.el8 rhel-8-for-x86_64-baseos-rpms 64 k 2023-01-24T16:38:49.682 INFO:teuthology.orchestra.run.smithi097.stdout: xz-devel x86_64 5.2.4-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2023-01-24T16:38:49.682 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:49.682 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-01-24T16:38:49.682 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-01-24T16:38:49.683 INFO:teuthology.orchestra.run.smithi097.stdout:Install 9 Packages 2023-01-24T16:38:49.683 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:49.684 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 1.9 M 2023-01-24T16:38:49.684 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 12 M 2023-01-24T16:38:49.684 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-01-24T16:38:49.729 INFO:teuthology.orchestra.run.smithi097.stdout:(1/9): xz-devel-5.2.4-3.el8.x86_64.rpm 1.3 MB/s | 62 kB 00:00 2023-01-24T16:38:49.731 INFO:teuthology.orchestra.run.smithi097.stdout:(2/9): libgpg-error-devel-1.31-1.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-01-24T16:38:49.739 INFO:teuthology.orchestra.run.smithi097.stdout:(3/9): libffi-devel-3.1-22.el8.x86_64.rpm 497 kB/s | 29 kB 00:00 2023-01-24T16:38:49.746 INFO:teuthology.orchestra.run.smithi097.stdout:(4/9): zlib-devel-1.2.11-17.el8.x86_64.rpm 3.8 MB/s | 58 kB 00:00 2023-01-24T16:38:49.777 INFO:teuthology.orchestra.run.smithi097.stdout:(5/9): libgcrypt-devel-1.8.5-4.el8.x86_64.rpm 3.1 MB/s | 149 kB 00:00 2023-01-24T16:38:49.779 INFO:teuthology.orchestra.run.smithi097.stdout:(6/9): libxslt-devel-1.1.32-6.el8.x86_64.rpm 9.7 MB/s | 323 kB 00:00 2023-01-24T16:38:49.786 INFO:teuthology.orchestra.run.smithi097.stdout:(7/9): libevent-devel-2.1.8-5.el8.x86_64.rpm 2.2 MB/s | 104 kB 00:00 2023-01-24T16:38:49.811 INFO:teuthology.orchestra.run.smithi097.stdout:(8/9): cmake-filesystem-3.18.2-11.el8_4.x86_64. 1.4 MB/s | 44 kB 00:00 2023-01-24T16:38:49.816 INFO:teuthology.orchestra.run.smithi097.stdout:(9/9): libxml2-devel-2.9.7-9.el8_4.2.x86_64.rpm 26 MB/s | 1.0 MB 00:00 2023-01-24T16:38:49.817 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-01-24T16:38:49.817 INFO:teuthology.orchestra.run.smithi097.stdout:Total 14 MB/s | 1.9 MB 00:00 2023-01-24T16:38:49.825 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-01-24T16:38:49.837 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-01-24T16:38:49.837 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-01-24T16:38:49.975 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-01-24T16:38:49.979 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-01-24T16:38:50.370 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-01-24T16:38:50.565 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 1/9 2023-01-24T16:38:50.594 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 1/9 2023-01-24T16:38:50.748 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgcrypt-devel-1.8.5-4.el8.x86_64 2/9 2023-01-24T16:38:50.760 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libgcrypt-devel-1.8.5-4.el8.x86_64 2/9 2023-01-24T16:38:50.947 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cmake-filesystem-3.18.2-11.el8_4.x86_64 3/9 2023-01-24T16:38:51.104 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 4/9 2023-01-24T16:38:51.537 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xz-devel-5.2.4-3.el8.x86_64 5/9 2023-01-24T16:38:51.889 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libxml2-devel-2.9.7-9.el8_4.2.x86_64 6/9 2023-01-24T16:38:52.130 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 7/9 2023-01-24T16:38:52.295 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 8/9 2023-01-24T16:38:52.447 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libffi-devel-3.1-22.el8.x86_64 9/9 2023-01-24T16:38:52.743 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libffi-devel-3.1-22.el8.x86_64 9/9 2023-01-24T16:38:52.743 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libffi-devel-3.1-22.el8.x86_64 1/9 2023-01-24T16:38:52.743 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xz-devel-5.2.4-3.el8.x86_64 2/9 2023-01-24T16:38:52.744 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 3/9 2023-01-24T16:38:52.744 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgcrypt-devel-1.8.5-4.el8.x86_64 4/9 2023-01-24T16:38:52.744 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 5/9 2023-01-24T16:38:52.744 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 6/9 2023-01-24T16:38:52.744 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 7/9 2023-01-24T16:38:52.745 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libxml2-devel-2.9.7-9.el8_4.2.x86_64 8/9 2023-01-24T16:38:53.751 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cmake-filesystem-3.18.2-11.el8_4.x86_64 9/9 2023-01-24T16:38:53.751 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:44 ago on Tue 24 Jan 2023 04:34:09 PM UTC. 2023-01-24T16:38:55.900 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-01-24T16:38:55.997 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:55.998 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-01-24T16:38:55.998 INFO:teuthology.orchestra.run.smithi097.stdout: cmake-filesystem-3.18.2-11.el8_4.x86_64 libevent-devel-2.1.8-5.el8.x86_64 2023-01-24T16:38:55.998 INFO:teuthology.orchestra.run.smithi097.stdout: libffi-devel-3.1-22.el8.x86_64 libgcrypt-devel-1.8.5-4.el8.x86_64 2023-01-24T16:38:55.998 INFO:teuthology.orchestra.run.smithi097.stdout: libgpg-error-devel-1.31-1.el8.x86_64 libxml2-devel-2.9.7-9.el8_4.2.x86_64 2023-01-24T16:38:55.998 INFO:teuthology.orchestra.run.smithi097.stdout: libxslt-devel-1.1.32-6.el8.x86_64 xz-devel-5.2.4-3.el8.x86_64 2023-01-24T16:38:55.999 INFO:teuthology.orchestra.run.smithi097.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-01-24T16:38:55.999 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:38:55.999 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-01-24T16:38:56.456 INFO:teuthology.orchestra.run.smithi097.stdout:Already using interpreter /usr/bin/python3.6 2023-01-24T16:38:56.456 INFO:teuthology.orchestra.run.smithi097.stdout:Using base prefix '/usr' 2023-01-24T16:38:56.456 INFO:teuthology.orchestra.run.smithi097.stdout:New python executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python3.6 2023-01-24T16:38:56.456 INFO:teuthology.orchestra.run.smithi097.stdout:Also creating executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python 2023-01-24T16:38:59.806 INFO:teuthology.orchestra.run.smithi097.stdout:Installing setuptools, pip, wheel...done. 2023-01-24T16:39:00.140 INFO:teuthology.orchestra.run.smithi097.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.6/site-packages (21.3.1) 2023-01-24T16:39:01.383 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting setuptools==44.1.0 2023-01-24T16:39:01.464 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2023-01-24T16:39:01.600 INFO:teuthology.orchestra.run.smithi097.stdout:Installing collected packages: setuptools 2023-01-24T16:39:01.600 INFO:teuthology.orchestra.run.smithi097.stdout: Attempting uninstall: setuptools 2023-01-24T16:39:01.600 INFO:teuthology.orchestra.run.smithi097.stdout: Found existing installation: setuptools 59.6.0 2023-01-24T16:39:01.643 INFO:teuthology.orchestra.run.smithi097.stdout: Uninstalling setuptools-59.6.0: 2023-01-24T16:39:01.651 INFO:teuthology.orchestra.run.smithi097.stdout: Successfully uninstalled setuptools-59.6.0 2023-01-24T16:39:01.918 INFO:teuthology.orchestra.run.smithi097.stdout:Successfully installed setuptools-44.1.0 2023-01-24T16:39:02.499 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting PyYAML 2023-01-24T16:39:02.590 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (603 kB) 2023-01-24T16:39:02.750 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting nose>=1.0.0 2023-01-24T16:39:02.778 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2023-01-24T16:39:02.905 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting boto>=2.6.0 2023-01-24T16:39:02.935 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2023-01-24T16:39:03.913 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting boto3>=1.0.0 2023-01-24T16:39:03.957 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2023-01-24T16:39:05.080 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting botocore<1.28.0 2023-01-24T16:39:05.133 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2023-01-24T16:39:05.794 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting munch>=2.0.0 2023-01-24T16:39:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading munch-2.5.0-py2.py3-none-any.whl (10 kB) 2023-01-24T16:39:06.773 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting gevent>=1.0 2023-01-24T16:39:06.803 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading gevent-22.10.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB) 2023-01-24T16:39:07.346 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting isodate>=0.4.4 2023-01-24T16:39:07.371 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2023-01-24T16:39:07.529 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting requests>=2.23.0 2023-01-24T16:39:07.556 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2023-01-24T16:39:07.818 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting pytz>=2011k 2023-01-24T16:39:07.882 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading pytz-2022.7.1-py2.py3-none-any.whl (499 kB) 2023-01-24T16:39:08.245 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting httplib2 2023-01-24T16:39:08.293 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading httplib2-0.21.0-py3-none-any.whl (96 kB) 2023-01-24T16:39:08.884 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting lxml 2023-01-24T16:39:08.923 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading lxml-4.9.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.6 MB) 2023-01-24T16:39:09.406 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting jmespath<2.0.0,>=0.7.1 2023-01-24T16:39:09.423 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2023-01-24T16:39:09.512 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2023-01-24T16:39:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2023-01-24T16:39:09.780 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting urllib3<1.27,>=1.25.4 2023-01-24T16:39:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB) 2023-01-24T16:39:09.906 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting python-dateutil<3.0.0,>=2.1 2023-01-24T16:39:09.923 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-01-24T16:39:10.028 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting six 2023-01-24T16:39:10.045 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-01-24T16:39:10.109 INFO:teuthology.orchestra.run.smithi097.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (from gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2023-01-24T16:39:10.479 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting zope.interface 2023-01-24T16:39:10.501 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253 kB) 2023-01-24T16:39:10.582 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting zope.event 2023-01-24T16:39:10.601 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB) 2023-01-24T16:39:10.967 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting greenlet>=2.0.0 2023-01-24T16:39:10.995 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading greenlet-2.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (513 kB) 2023-01-24T16:39:11.248 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting charset-normalizer~=2.0.0 2023-01-24T16:39:11.268 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2023-01-24T16:39:11.387 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting certifi>=2017.4.17 2023-01-24T16:39:11.404 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading certifi-2022.12.7-py3-none-any.whl (155 kB) 2023-01-24T16:39:11.584 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting idna<4,>=2.5 2023-01-24T16:39:11.601 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading idna-3.4-py3-none-any.whl (61 kB) 2023-01-24T16:39:11.748 INFO:teuthology.orchestra.run.smithi097.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2023-01-24T16:39:11.764 INFO:teuthology.orchestra.run.smithi097.stdout: Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB) 2023-01-24T16:39:12.335 INFO:teuthology.orchestra.run.smithi097.stdout:Installing collected packages: six, urllib3, python-dateutil, jmespath, botocore, zope.interface, zope.event, s3transfer, pyparsing, idna, greenlet, charset-normalizer, certifi, requests, PyYAML, pytz, nose, munch, lxml, isodate, httplib2, gevent, boto3, boto 2023-01-24T16:39:15.849 INFO:teuthology.orchestra.run.smithi097.stdout:Successfully installed PyYAML-6.0 boto-2.49.0 boto3-1.23.10 botocore-1.26.10 certifi-2022.12.7 charset-normalizer-2.0.12 gevent-22.10.2 greenlet-2.0.1 httplib2-0.21.0 idna-3.4 isodate-0.6.1 jmespath-0.10.0 lxml-4.9.2 munch-2.5.0 nose-1.3.7 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2022.7.1 requests-2.27.1 s3transfer-0.5.2 six-1.16.0 urllib3-1.26.14 zope.event-4.6 zope.interface-5.5.2 2023-01-24T16:39:16.141 INFO:teuthology.orchestra.run.smithi097.stdout:running develop 2023-01-24T16:39:16.160 INFO:teuthology.orchestra.run.smithi097.stdout:running egg_info 2023-01-24T16:39:16.160 INFO:teuthology.orchestra.run.smithi097.stdout:creating s3tests.egg-info 2023-01-24T16:39:16.161 INFO:teuthology.orchestra.run.smithi097.stdout:writing s3tests.egg-info/PKG-INFO 2023-01-24T16:39:16.161 INFO:teuthology.orchestra.run.smithi097.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2023-01-24T16:39:16.161 INFO:teuthology.orchestra.run.smithi097.stdout:writing requirements to s3tests.egg-info/requires.txt 2023-01-24T16:39:16.161 INFO:teuthology.orchestra.run.smithi097.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2023-01-24T16:39:16.161 INFO:teuthology.orchestra.run.smithi097.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-01-24T16:39:16.170 INFO:teuthology.orchestra.run.smithi097.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2023-01-24T16:39:16.172 INFO:teuthology.orchestra.run.smithi097.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-01-24T16:39:16.184 INFO:teuthology.orchestra.run.smithi097.stdout:running build_ext 2023-01-24T16:39:16.184 INFO:teuthology.orchestra.run.smithi097.stdout:Creating /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/s3tests.egg-link (link to .) 2023-01-24T16:39:16.184 INFO:teuthology.orchestra.run.smithi097.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2023-01-24T16:39:16.185 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.185 INFO:teuthology.orchestra.run.smithi097.stdout:Installed /home/ubuntu/cephtest/s3-tests 2023-01-24T16:39:16.186 INFO:teuthology.orchestra.run.smithi097.stdout:Processing dependencies for s3tests==0.0.1 2023-01-24T16:39:16.274 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for isodate==0.6.1 2023-01-24T16:39:16.274 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: isodate 0.6.1 2023-01-24T16:39:16.274 INFO:teuthology.orchestra.run.smithi097.stdout:Adding isodate 0.6.1 to easy-install.pth file 2023-01-24T16:39:16.274 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.275 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.276 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for gevent==22.10.2 2023-01-24T16:39:16.276 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: gevent 22.10.2 2023-01-24T16:39:16.276 INFO:teuthology.orchestra.run.smithi097.stdout:Adding gevent 22.10.2 to easy-install.pth file 2023-01-24T16:39:16.276 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.277 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.277 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for munch==2.5.0 2023-01-24T16:39:16.277 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: munch 2.5.0 2023-01-24T16:39:16.277 INFO:teuthology.orchestra.run.smithi097.stdout:Adding munch 2.5.0 to easy-install.pth file 2023-01-24T16:39:16.277 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.278 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.278 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for PyYAML==6.0 2023-01-24T16:39:16.278 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: PyYAML 6.0 2023-01-24T16:39:16.280 INFO:teuthology.orchestra.run.smithi097.stdout:Adding PyYAML 6.0 to easy-install.pth file 2023-01-24T16:39:16.280 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.281 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.281 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for boto3==1.23.10 2023-01-24T16:39:16.281 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: boto3 1.23.10 2023-01-24T16:39:16.281 INFO:teuthology.orchestra.run.smithi097.stdout:Adding boto3 1.23.10 to easy-install.pth file 2023-01-24T16:39:16.281 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.282 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.282 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for boto==2.49.0 2023-01-24T16:39:16.282 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: boto 2.49.0 2023-01-24T16:39:16.282 INFO:teuthology.orchestra.run.smithi097.stdout:Adding boto 2.49.0 to easy-install.pth file 2023-01-24T16:39:16.282 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.283 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.283 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for six==1.16.0 2023-01-24T16:39:16.283 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: six 1.16.0 2023-01-24T16:39:16.283 INFO:teuthology.orchestra.run.smithi097.stdout:Adding six 1.16.0 to easy-install.pth file 2023-01-24T16:39:16.283 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.283 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.284 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for greenlet==2.0.1 2023-01-24T16:39:16.284 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: greenlet 2.0.1 2023-01-24T16:39:16.284 INFO:teuthology.orchestra.run.smithi097.stdout:Adding greenlet 2.0.1 to easy-install.pth file 2023-01-24T16:39:16.284 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.284 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.285 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for zope.event==4.6 2023-01-24T16:39:16.285 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: zope.event 4.6 2023-01-24T16:39:16.285 INFO:teuthology.orchestra.run.smithi097.stdout:Adding zope.event 4.6 to easy-install.pth file 2023-01-24T16:39:16.285 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.285 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.288 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for zope.interface==5.5.2 2023-01-24T16:39:16.289 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: zope.interface 5.5.2 2023-01-24T16:39:16.289 INFO:teuthology.orchestra.run.smithi097.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2023-01-24T16:39:16.289 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.289 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.290 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for setuptools==44.1.0 2023-01-24T16:39:16.290 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: setuptools 44.1.0 2023-01-24T16:39:16.290 INFO:teuthology.orchestra.run.smithi097.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2023-01-24T16:39:16.290 INFO:teuthology.orchestra.run.smithi097.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2023-01-24T16:39:16.290 INFO:teuthology.orchestra.run.smithi097.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2023-01-24T16:39:16.290 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.291 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.291 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for s3transfer==0.5.2 2023-01-24T16:39:16.291 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: s3transfer 0.5.2 2023-01-24T16:39:16.291 INFO:teuthology.orchestra.run.smithi097.stdout:Adding s3transfer 0.5.2 to easy-install.pth file 2023-01-24T16:39:16.291 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.292 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.292 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for jmespath==0.10.0 2023-01-24T16:39:16.292 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: jmespath 0.10.0 2023-01-24T16:39:16.292 INFO:teuthology.orchestra.run.smithi097.stdout:Adding jmespath 0.10.0 to easy-install.pth file 2023-01-24T16:39:16.293 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.293 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.293 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for botocore==1.26.10 2023-01-24T16:39:16.293 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: botocore 1.26.10 2023-01-24T16:39:16.293 INFO:teuthology.orchestra.run.smithi097.stdout:Adding botocore 1.26.10 to easy-install.pth file 2023-01-24T16:39:16.294 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.294 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.294 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for urllib3==1.26.14 2023-01-24T16:39:16.294 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: urllib3 1.26.14 2023-01-24T16:39:16.294 INFO:teuthology.orchestra.run.smithi097.stdout:Adding urllib3 1.26.14 to easy-install.pth file 2023-01-24T16:39:16.295 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.295 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.295 INFO:teuthology.orchestra.run.smithi097.stdout:Searching for python-dateutil==2.8.2 2023-01-24T16:39:16.295 INFO:teuthology.orchestra.run.smithi097.stdout:Best match: python-dateutil 2.8.2 2023-01-24T16:39:16.295 INFO:teuthology.orchestra.run.smithi097.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2023-01-24T16:39:16.296 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:39:16.296 INFO:teuthology.orchestra.run.smithi097.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T16:39:16.296 INFO:teuthology.orchestra.run.smithi097.stdout:Finished processing dependencies for s3tests==0.0.1 2023-01-24T16:39:16.307 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:39:16.308 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2023-01-24T16:39:16.321 INFO:tasks.s3tests:Configuring boto... 2023-01-24T16:39:16.322 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-01-24T16:39:16.322 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/boto.cfg 2023-01-24T16:39:16.376 DEBUG:teuthology.orchestra.run.smithi097:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a lifecycle 2023-01-24T16:39:17.114 INFO:teuthology.orchestra.run.smithi097.stderr:/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2023-01-24T16:39:17.115 INFO:teuthology.orchestra.run.smithi097.stderr: warnings.warn(warning, PythonDeprecationWarning) 2023-01-24T16:39:17.145 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2023-01-24T16:39:17.215 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2023-01-24T16:39:17.289 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2023-01-24T16:40:27.505 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration ... ok 2023-01-24T16:41:37.687 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecyclev2_expiration ... ok 2023-01-24T16:42:07.836 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_versioning_enabled ... ok 2023-01-24T16:42:37.909 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_tags1 ... ok 2023-01-24T16:43:08.018 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_tags2 ... ok 2023-01-24T16:43:38.186 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_versioned_tags2 ... ok 2023-01-24T16:44:48.525 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_noncur_tags1 ... ok 2023-01-24T16:44:48.553 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2023-01-24T16:44:48.583 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2023-01-24T16:44:48.697 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2023-01-24T16:44:48.734 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2023-01-24T16:44:48.765 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2023-01-24T16:45:18.907 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_date ... ok 2023-01-24T16:45:18.962 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_days0 ... ok 2023-01-24T16:45:19.030 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_header_put ... ok 2023-01-24T16:45:19.069 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_header_head ... ok 2023-01-24T16:45:19.135 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_header_tags_head ... ok 2023-01-24T16:45:19.194 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_header_and_tags_head ... ok 2023-01-24T16:45:19.238 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2023-01-24T16:46:09.477 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_noncur_expiration ... ok 2023-01-24T16:46:09.511 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2023-01-24T16:46:09.548 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2023-01-24T16:46:09.595 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2023-01-24T16:47:19.814 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_deletemarker_expiration ... ok 2023-01-24T16:47:19.850 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2023-01-24T16:48:11.203 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_multipart_expiration ... ok 2023-01-24T16:48:11.238 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_set_invalid_date ... ok 2023-01-24T16:49:51.474 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition ... ok 2023-01-24T16:51:51.721 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... ok 2023-01-24T16:51:51.757 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent_transition ... ok 2023-01-24T16:52:31.983 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_noncur_transition ... FAIL 2023-01-24T16:52:31.984 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition ... SKIP 2023-01-24T16:52:31.984 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_multiple_transition ... SKIP 2023-01-24T16:52:31.984 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_noncur_cloud_transition ... SKIP 2023-01-24T16:52:35.427 INFO:teuthology.orchestra.run.smithi097.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition_large_obj ... SKIP 2023-01-24T16:52:35.427 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.428 INFO:teuthology.orchestra.run.smithi097.stderr:====================================================================== 2023-01-24T16:52:35.428 INFO:teuthology.orchestra.run.smithi097.stderr:FAIL: s3tests_boto3.functional.test_s3.test_lifecycle_noncur_transition 2023-01-24T16:52:35.428 INFO:teuthology.orchestra.run.smithi097.stderr:---------------------------------------------------------------------- 2023-01-24T16:52:35.428 INFO:teuthology.orchestra.run.smithi097.stderr:Traceback (most recent call last): 2023-01-24T16:52:35.429 INFO:teuthology.orchestra.run.smithi097.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2023-01-24T16:52:35.429 INFO:teuthology.orchestra.run.smithi097.stderr: self.test(*self.arg) 2023-01-24T16:52:35.429 INFO:teuthology.orchestra.run.smithi097.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 10344, in test_lifecycle_noncur_transition 2023-01-24T16:52:35.429 INFO:teuthology.orchestra.run.smithi097.stderr: eq(len(expire1_keys['STANDARD']), 2) 2023-01-24T16:52:35.429 INFO:teuthology.orchestra.run.smithi097.stderr:AssertionError: 6 != 2 2023-01-24T16:52:35.430 INFO:teuthology.orchestra.run.smithi097.stderr:-------------------- >> begin captured stdout << --------------------- 2023-01-24T16:52:35.430 INFO:teuthology.orchestra.run.smithi097.stderr:storage classes configured: ['STANDARD', 'LUKEWARM', 'FROZEN'] 2023-01-24T16:52:35.430 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.430 INFO:teuthology.orchestra.run.smithi097.stderr:--------------------- >> end captured stdout << ---------------------- 2023-01-24T16:52:35.430 INFO:teuthology.orchestra.run.smithi097.stderr:-------------------- >> begin captured logging << -------------------- 2023-01-24T16:52:35.431 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2023-01-24T16:52:35.431 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2023-01-24T16:52:35.431 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f26eb298f28> 2023-01-24T16:52:35.431 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2023-01-24T16:52:35.431 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2023-01-24T16:52:35.432 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2023-01-24T16:52:35.432 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2023-01-24T16:52:35.432 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2023-01-24T16:52:35.432 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2023-01-24T16:52:35.432 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2023-01-24T16:52:35.433 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-01-24T16:52:35.433 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2023-01-24T16:52:35.433 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-01-24T16:52:35.433 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-01-24T16:52:35.434 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-767ump700z8ujae-34', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-305.25.1.el8_4.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi097.front.sepia.ceph.com:80/test-client.0-767ump700z8ujae-34', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-767ump700z8ujae-34'}}} 2023-01-24T16:52:35.434 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2023-01-24T16:52:35.434 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2023-01-24T16:52:35.434 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2023-01-24T16:52:35.434 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2023-01-24T16:52:35.435 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2023-01-24T16:52:35.435 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-01-24T16:52:35.435 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-01-24T16:52:35.435 INFO:teuthology.orchestra.run.smithi097.stderr:PUT 2023-01-24T16:52:35.435 INFO:teuthology.orchestra.run.smithi097.stderr:/test-client.0-767ump700z8ujae-34 2023-01-24T16:52:35.436 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.436 INFO:teuthology.orchestra.run.smithi097.stderr:host:smithi097.front.sepia.ceph.com 2023-01-24T16:52:35.436 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-01-24T16:52:35.436 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-date:20230124T165151Z 2023-01-24T16:52:35.436 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.437 INFO:teuthology.orchestra.run.smithi097.stderr:host;x-amz-content-sha256;x-amz-date 2023-01-24T16:52:35.437 INFO:teuthology.orchestra.run.smithi097.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-01-24T16:52:35.437 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: StringToSign: 2023-01-24T16:52:35.437 INFO:teuthology.orchestra.run.smithi097.stderr:AWS4-HMAC-SHA256 2023-01-24T16:52:35.437 INFO:teuthology.orchestra.run.smithi097.stderr:20230124T165151Z 2023-01-24T16:52:35.438 INFO:teuthology.orchestra.run.smithi097.stderr:20230124/us-east-1/s3/aws4_request 2023-01-24T16:52:35.438 INFO:teuthology.orchestra.run.smithi097.stderr:ecf1802dfad075e7078cfb2514939ea1144b3380c3d1ff811b911497bf522cba 2023-01-24T16:52:35.438 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Signature: 2023-01-24T16:52:35.438 INFO:teuthology.orchestra.run.smithi097.stderr:fd46d203f9defb553237f88847ab45d0ea166babb00f389a09915866a36c3d4b 2023-01-24T16:52:35.438 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2023-01-24T16:52:35.438 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-01-24T16:52:35.439 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi097.front.sepia.ceph.com:80 2023-01-24T16:52:35.439 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: http://smithi097.front.sepia.ceph.com:80 "PUT /test-client.0-767ump700z8ujae-34 HTTP/1.1" 200 0 2023-01-24T16:52:35.439 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx0000036bababcdf896d11-0063d00ca7-1089-default', 'Content-Length': '0', 'Date': 'Tue, 24 Jan 2023 16:51:51 GMT', 'Connection': 'Keep-Alive'} 2023-01-24T16:52:35.439 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response body: 2023-01-24T16:52:35.439 INFO:teuthology.orchestra.run.smithi097.stderr:b'' 2023-01-24T16:52:35.440 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2023-01-24T16:52:35.440 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-01-24T16:52:35.440 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2023-01-24T16:52:35.440 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2023-01-24T16:52:35.440 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2023-01-24T16:52:35.441 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f26eb298f28> 2023-01-24T16:52:35.441 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2023-01-24T16:52:35.441 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2023-01-24T16:52:35.441 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2023-01-24T16:52:35.441 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2023-01-24T16:52:35.442 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2023-01-24T16:52:35.442 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f26eb298f28> 2023-01-24T16:52:35.442 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2023-01-24T16:52:35.442 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2023-01-24T16:52:35.442 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2023-01-24T16:52:35.443 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketVersioning: calling handler 2023-01-24T16:52:35.443 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketVersioning: calling handler > 2023-01-24T16:52:35.443 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketVersioning: calling handler > 2023-01-24T16:52:35.443 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketVersioning: calling handler 2023-01-24T16:52:35.443 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketVersioning: calling handler 2023-01-24T16:52:35.444 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketVersioning: calling handler > 2023-01-24T16:52:35.444 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketVersioning: calling handler 2023-01-24T16:52:35.444 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketVersioning: calling handler 2023-01-24T16:52:35.444 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=PutBucketVersioning) with params: {'url_path': '/test-client.0-767ump700z8ujae-34?versioning', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-305.25.1.el8_4.x86_64 Botocore/1.26.10', 'Content-MD5': 'QQFYoy/mRYV9PGZUfFi0Bw=='}, 'body': b'Enabled', 'url': 'http://smithi097.front.sepia.ceph.com:80/test-client.0-767ump700z8ujae-34?versioning', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-767ump700z8ujae-34'}, 'checksum': {'request_algorithm': 'conditional-md5'}}} 2023-01-24T16:52:35.444 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutBucketVersioning: calling handler > 2023-01-24T16:52:35.445 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutBucketVersioning: calling handler > 2023-01-24T16:52:35.445 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutBucketVersioning: calling handler 2023-01-24T16:52:35.445 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-sign.s3.PutBucketVersioning: calling handler > 2023-01-24T16:52:35.445 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2023-01-24T16:52:35.445 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-01-24T16:52:35.445 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-01-24T16:52:35.446 INFO:teuthology.orchestra.run.smithi097.stderr:PUT 2023-01-24T16:52:35.446 INFO:teuthology.orchestra.run.smithi097.stderr:/test-client.0-767ump700z8ujae-34 2023-01-24T16:52:35.446 INFO:teuthology.orchestra.run.smithi097.stderr:versioning= 2023-01-24T16:52:35.446 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5:QQFYoy/mRYV9PGZUfFi0Bw== 2023-01-24T16:52:35.446 INFO:teuthology.orchestra.run.smithi097.stderr:host:smithi097.front.sepia.ceph.com 2023-01-24T16:52:35.447 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-content-sha256:482d4e510b5cdcf0bbf3a044e82832dc711f4f28757a6c06eb21758282a73ef7 2023-01-24T16:52:35.447 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-date:20230124T165151Z 2023-01-24T16:52:35.447 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.447 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5;host;x-amz-content-sha256;x-amz-date 2023-01-24T16:52:35.447 INFO:teuthology.orchestra.run.smithi097.stderr:482d4e510b5cdcf0bbf3a044e82832dc711f4f28757a6c06eb21758282a73ef7 2023-01-24T16:52:35.448 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: StringToSign: 2023-01-24T16:52:35.448 INFO:teuthology.orchestra.run.smithi097.stderr:AWS4-HMAC-SHA256 2023-01-24T16:52:35.448 INFO:teuthology.orchestra.run.smithi097.stderr:20230124T165151Z 2023-01-24T16:52:35.448 INFO:teuthology.orchestra.run.smithi097.stderr:20230124/us-east-1/s3/aws4_request 2023-01-24T16:52:35.448 INFO:teuthology.orchestra.run.smithi097.stderr:251e092abaf3259114113252088fbe04ca044cdff1b8177f15af570d741aaad8 2023-01-24T16:52:35.449 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Signature: 2023-01-24T16:52:35.449 INFO:teuthology.orchestra.run.smithi097.stderr:7a08c598960e3e123a988fda87b5840bb2c957c522c36f9b96d799345c468346 2023-01-24T16:52:35.449 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutBucketVersioning: calling handler 2023-01-24T16:52:35.449 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-01-24T16:52:35.449 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi097.front.sepia.ceph.com:80 2023-01-24T16:52:35.450 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: http://smithi097.front.sepia.ceph.com:80 "PUT /test-client.0-767ump700z8ujae-34?versioning HTTP/1.1" 200 0 2023-01-24T16:52:35.450 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000c35058fe92c74de5-0063d00ca7-1089-default', 'Content-Type': 'application/xml', 'Content-Length': '0', 'Date': 'Tue, 24 Jan 2023 16:51:51 GMT', 'Connection': 'Keep-Alive'} 2023-01-24T16:52:35.450 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response body: 2023-01-24T16:52:35.450 INFO:teuthology.orchestra.run.smithi097.stderr:b'' 2023-01-24T16:52:35.450 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutBucketVersioning: calling handler 2023-01-24T16:52:35.451 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-01-24T16:52:35.451 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutBucketVersioning: calling handler > 2023-01-24T16:52:35.451 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.GetBucketVersioning: calling handler 2023-01-24T16:52:35.451 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.GetBucketVersioning: calling handler > 2023-01-24T16:52:35.451 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.GetBucketVersioning: calling handler > 2023-01-24T16:52:35.452 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.GetBucketVersioning: calling handler 2023-01-24T16:52:35.452 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.GetBucketVersioning: calling handler 2023-01-24T16:52:35.452 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.GetBucketVersioning: calling handler > 2023-01-24T16:52:35.452 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.GetBucketVersioning: calling handler 2023-01-24T16:52:35.452 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.GetBucketVersioning: calling handler 2023-01-24T16:52:35.453 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=GetBucketVersioning) with params: {'url_path': '/test-client.0-767ump700z8ujae-34?versioning', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-305.25.1.el8_4.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi097.front.sepia.ceph.com:80/test-client.0-767ump700z8ujae-34?versioning', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-767ump700z8ujae-34'}}} 2023-01-24T16:52:35.453 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.GetBucketVersioning: calling handler > 2023-01-24T16:52:35.453 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.GetBucketVersioning: calling handler > 2023-01-24T16:52:35.454 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.GetBucketVersioning: calling handler 2023-01-24T16:52:35.454 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-sign.s3.GetBucketVersioning: calling handler > 2023-01-24T16:52:35.454 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-01-24T16:52:35.454 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-01-24T16:52:35.454 INFO:teuthology.orchestra.run.smithi097.stderr:GET 2023-01-24T16:52:35.455 INFO:teuthology.orchestra.run.smithi097.stderr:/test-client.0-767ump700z8ujae-34 2023-01-24T16:52:35.455 INFO:teuthology.orchestra.run.smithi097.stderr:versioning= 2023-01-24T16:52:35.455 INFO:teuthology.orchestra.run.smithi097.stderr:host:smithi097.front.sepia.ceph.com 2023-01-24T16:52:35.455 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-01-24T16:52:35.455 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-date:20230124T165151Z 2023-01-24T16:52:35.455 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.456 INFO:teuthology.orchestra.run.smithi097.stderr:host;x-amz-content-sha256;x-amz-date 2023-01-24T16:52:35.456 INFO:teuthology.orchestra.run.smithi097.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-01-24T16:52:35.456 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: StringToSign: 2023-01-24T16:52:35.456 INFO:teuthology.orchestra.run.smithi097.stderr:AWS4-HMAC-SHA256 2023-01-24T16:52:35.457 INFO:teuthology.orchestra.run.smithi097.stderr:20230124T165151Z 2023-01-24T16:52:35.457 INFO:teuthology.orchestra.run.smithi097.stderr:20230124/us-east-1/s3/aws4_request 2023-01-24T16:52:35.457 INFO:teuthology.orchestra.run.smithi097.stderr:1f2aecc3773b0e921e9f019c151cb75f44afa187978f5cba78dd295ca1481924 2023-01-24T16:52:35.457 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Signature: 2023-01-24T16:52:35.457 INFO:teuthology.orchestra.run.smithi097.stderr:d5072da0dc75c8d306ebd31b2524d583c086660072e1d23e855477c89597625c 2023-01-24T16:52:35.458 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.GetBucketVersioning: calling handler 2023-01-24T16:52:35.458 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-01-24T16:52:35.458 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: http://smithi097.front.sepia.ceph.com:80 "GET /test-client.0-767ump700z8ujae-34?versioning HTTP/1.1" 200 192 2023-01-24T16:52:35.458 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000f624634e4559aece-0063d00ca7-1089-default', 'Content-Type': 'application/xml', 'Content-Length': '192', 'Date': 'Tue, 24 Jan 2023 16:51:51 GMT', 'Connection': 'Keep-Alive'} 2023-01-24T16:52:35.458 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response body: 2023-01-24T16:52:35.458 INFO:teuthology.orchestra.run.smithi097.stderr:b'EnabledDisabled' 2023-01-24T16:52:35.459 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.GetBucketVersioning: calling handler 2023-01-24T16:52:35.459 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-01-24T16:52:35.459 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.GetBucketVersioning: calling handler > 2023-01-24T16:52:35.459 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketLifecycleConfiguration: calling handler 2023-01-24T16:52:35.460 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketLifecycleConfiguration: calling handler > 2023-01-24T16:52:35.460 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketLifecycleConfiguration: calling handler > 2023-01-24T16:52:35.460 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketLifecycleConfiguration: calling handler 2023-01-24T16:52:35.460 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketLifecycleConfiguration: calling handler 2023-01-24T16:52:35.460 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketLifecycleConfiguration: calling handler 2023-01-24T16:52:35.461 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketLifecycleConfiguration: calling handler > 2023-01-24T16:52:35.461 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketLifecycleConfiguration: calling handler 2023-01-24T16:52:35.461 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketLifecycleConfiguration: calling handler 2023-01-24T16:52:35.461 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=PutBucketLifecycleConfiguration) with params: {'url_path': '/test-client.0-767ump700z8ujae-34?lifecycle', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-305.25.1.el8_4.x86_64 Botocore/1.26.10', 'Content-MD5': 'cQs3aEDRr4FSQzrW0tz5Bg=='}, 'body': b'rule1test1/Enabled1LUKEWARM5FROZEN9', 'url': 'http://smithi097.front.sepia.ceph.com:80/test-client.0-767ump700z8ujae-34?lifecycle', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-767ump700z8ujae-34'}, 'checksum': {'request_algorithm': 'conditional-md5'}}} 2023-01-24T16:52:35.461 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutBucketLifecycleConfiguration: calling handler > 2023-01-24T16:52:35.462 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutBucketLifecycleConfiguration: calling handler > 2023-01-24T16:52:35.462 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutBucketLifecycleConfiguration: calling handler 2023-01-24T16:52:35.462 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-sign.s3.PutBucketLifecycleConfiguration: calling handler > 2023-01-24T16:52:35.462 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2023-01-24T16:52:35.462 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-01-24T16:52:35.463 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-01-24T16:52:35.463 INFO:teuthology.orchestra.run.smithi097.stderr:PUT 2023-01-24T16:52:35.463 INFO:teuthology.orchestra.run.smithi097.stderr:/test-client.0-767ump700z8ujae-34 2023-01-24T16:52:35.463 INFO:teuthology.orchestra.run.smithi097.stderr:lifecycle= 2023-01-24T16:52:35.463 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5:cQs3aEDRr4FSQzrW0tz5Bg== 2023-01-24T16:52:35.464 INFO:teuthology.orchestra.run.smithi097.stderr:host:smithi097.front.sepia.ceph.com 2023-01-24T16:52:35.464 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-content-sha256:accf437885cbcff9eea7a49a41a4db2a09288a4daf1ea71f6c6527d33921b307 2023-01-24T16:52:35.464 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-date:20230124T165151Z 2023-01-24T16:52:35.464 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.464 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5;host;x-amz-content-sha256;x-amz-date 2023-01-24T16:52:35.465 INFO:teuthology.orchestra.run.smithi097.stderr:accf437885cbcff9eea7a49a41a4db2a09288a4daf1ea71f6c6527d33921b307 2023-01-24T16:52:35.465 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: StringToSign: 2023-01-24T16:52:35.465 INFO:teuthology.orchestra.run.smithi097.stderr:AWS4-HMAC-SHA256 2023-01-24T16:52:35.465 INFO:teuthology.orchestra.run.smithi097.stderr:20230124T165151Z 2023-01-24T16:52:35.465 INFO:teuthology.orchestra.run.smithi097.stderr:20230124/us-east-1/s3/aws4_request 2023-01-24T16:52:35.466 INFO:teuthology.orchestra.run.smithi097.stderr:92623f7a1dc0b91db166e0936e7eaec08854a3e7aabb034a283214221cc6ad06 2023-01-24T16:52:35.466 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Signature: 2023-01-24T16:52:35.466 INFO:teuthology.orchestra.run.smithi097.stderr:558de0cbb42fa5307a2b9de46a885d0ad51a7de8b0d0542fe415fdf22e90fef5 2023-01-24T16:52:35.466 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutBucketLifecycleConfiguration: calling handler 2023-01-24T16:52:35.466 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-01-24T16:52:35.466 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi097.front.sepia.ceph.com:80 2023-01-24T16:52:35.467 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: http://smithi097.front.sepia.ceph.com:80 "PUT /test-client.0-767ump700z8ujae-34?lifecycle HTTP/1.1" 200 0 2023-01-24T16:52:35.467 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000de956a4a4c984dd6-0063d00ca7-1089-default', 'Content-Type': 'application/xml', 'Content-Length': '0', 'Date': 'Tue, 24 Jan 2023 16:51:51 GMT', 'Connection': 'Keep-Alive'} 2023-01-24T16:52:35.467 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response body: 2023-01-24T16:52:35.467 INFO:teuthology.orchestra.run.smithi097.stderr:b'' 2023-01-24T16:52:35.467 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutBucketLifecycleConfiguration: calling handler 2023-01-24T16:52:35.468 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-01-24T16:52:35.468 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutBucketLifecycleConfiguration: calling handler > 2023-01-24T16:52:35.468 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.468 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.468 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.469 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.469 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler > 2023-01-24T16:52:35.469 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler > 2023-01-24T16:52:35.469 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.469 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.470 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.470 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.handlers: DEBUG: Adding expect 100 continue header to request. 2023-01-24T16:52:35.470 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler > 2023-01-24T16:52:35.470 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.470 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.470 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=PutObject) with params: {'url_path': '/test-client.0-767ump700z8ujae-34/test1/a', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-305.25.1.el8_4.x86_64 Botocore/1.26.10', 'Content-MD5': 'jmibqIuS5AIMZQjopz5OwQ==', 'Expect': '100-continue'}, 'body': <_io.BytesIO object at 0x7f26ecc1d4c0>, 'url': 'http://smithi097.front.sepia.ceph.com:80/test-client.0-767ump700z8ujae-34/test1/a', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': True, 'auth_type': None, 'signing': {'bucket': 'test-client.0-767ump700z8ujae-34'}}} 2023-01-24T16:52:35.471 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutObject: calling handler > 2023-01-24T16:52:35.471 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler > 2023-01-24T16:52:35.471 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler 2023-01-24T16:52:35.471 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-sign.s3.PutObject: calling handler > 2023-01-24T16:52:35.471 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-01-24T16:52:35.472 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-01-24T16:52:35.472 INFO:teuthology.orchestra.run.smithi097.stderr:PUT 2023-01-24T16:52:35.472 INFO:teuthology.orchestra.run.smithi097.stderr:/test-client.0-767ump700z8ujae-34/test1/a 2023-01-24T16:52:35.472 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.472 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5:jmibqIuS5AIMZQjopz5OwQ== 2023-01-24T16:52:35.473 INFO:teuthology.orchestra.run.smithi097.stderr:host:smithi097.front.sepia.ceph.com 2023-01-24T16:52:35.473 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-content-sha256:913bea5c2f5b94d7eede7a4ddec9110d312682917cbc114eb4f729a26428d02c 2023-01-24T16:52:35.473 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-date:20230124T165151Z 2023-01-24T16:52:35.473 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.473 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5;host;x-amz-content-sha256;x-amz-date 2023-01-24T16:52:35.473 INFO:teuthology.orchestra.run.smithi097.stderr:913bea5c2f5b94d7eede7a4ddec9110d312682917cbc114eb4f729a26428d02c 2023-01-24T16:52:35.474 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: StringToSign: 2023-01-24T16:52:35.474 INFO:teuthology.orchestra.run.smithi097.stderr:AWS4-HMAC-SHA256 2023-01-24T16:52:35.474 INFO:teuthology.orchestra.run.smithi097.stderr:20230124T165151Z 2023-01-24T16:52:35.474 INFO:teuthology.orchestra.run.smithi097.stderr:20230124/us-east-1/s3/aws4_request 2023-01-24T16:52:35.474 INFO:teuthology.orchestra.run.smithi097.stderr:4c9ca3a1be5427d0b2b63084e4e1dd58354670f70ba328daf019e7c6f0e16f1f 2023-01-24T16:52:35.475 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Signature: 2023-01-24T16:52:35.475 INFO:teuthology.orchestra.run.smithi097.stderr:52988105149bb56efa9bffbddb221013e3743bd1f7a92fc56129043b58c813db 2023-01-24T16:52:35.475 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutObject: calling handler 2023-01-24T16:52:35.475 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-01-24T16:52:35.475 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.awsrequest: DEBUG: Waiting for 100 Continue response. 2023-01-24T16:52:35.476 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.awsrequest: DEBUG: 100 Continue response seen, now sending request body. 2023-01-24T16:52:35.476 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: http://smithi097.front.sepia.ceph.com:80 "PUT /test-client.0-767ump700z8ujae-34/test1/a HTTP/1.1" 200 0 2023-01-24T16:52:35.476 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response headers: {'Content-Length': '0', 'ETag': '"8e689ba88b92e4020c6508e8a73e4ec1"', 'Accept-Ranges': 'bytes', 'x-amz-version-id': 'BFAGISa0bubO4Xh-r1MB2nv8usJk24z', 'x-amz-expiration': 'expiry-date="Fri, 03 Feb 2023 00:00:00 GMT", rule-id="rule1"', 'x-amz-request-id': 'tx00000069419d2b00676be-0063d00ca7-1089-default', 'Date': 'Tue, 24 Jan 2023 16:51:51 GMT', 'Connection': 'Keep-Alive'} 2023-01-24T16:52:35.476 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response body: 2023-01-24T16:52:35.476 INFO:teuthology.orchestra.run.smithi097.stderr:b'' 2023-01-24T16:52:35.477 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutObject: calling handler 2023-01-24T16:52:35.477 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-01-24T16:52:35.477 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutObject: calling handler > 2023-01-24T16:52:35.477 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.477 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.477 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.478 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.478 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler > 2023-01-24T16:52:35.478 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler > 2023-01-24T16:52:35.478 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.478 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.479 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.479 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.handlers: DEBUG: Adding expect 100 continue header to request. 2023-01-24T16:52:35.479 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler > 2023-01-24T16:52:35.479 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.479 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.480 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=PutObject) with params: {'url_path': '/test-client.0-767ump700z8ujae-34/test1/a', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-305.25.1.el8_4.x86_64 Botocore/1.26.10', 'Content-MD5': 'koyuYjFLlVJ5GO5KJEfaAQ==', 'Expect': '100-continue'}, 'body': <_io.BytesIO object at 0x7f26ecc1d830>, 'url': 'http://smithi097.front.sepia.ceph.com:80/test-client.0-767ump700z8ujae-34/test1/a', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': True, 'auth_type': None, 'signing': {'bucket': 'test-client.0-767ump700z8ujae-34'}}} 2023-01-24T16:52:35.480 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutObject: calling handler > 2023-01-24T16:52:35.480 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler > 2023-01-24T16:52:35.480 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler 2023-01-24T16:52:35.480 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-sign.s3.PutObject: calling handler > 2023-01-24T16:52:35.481 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-01-24T16:52:35.481 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-01-24T16:52:35.481 INFO:teuthology.orchestra.run.smithi097.stderr:PUT 2023-01-24T16:52:35.481 INFO:teuthology.orchestra.run.smithi097.stderr:/test-client.0-767ump700z8ujae-34/test1/a 2023-01-24T16:52:35.481 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.481 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5:koyuYjFLlVJ5GO5KJEfaAQ== 2023-01-24T16:52:35.482 INFO:teuthology.orchestra.run.smithi097.stderr:host:smithi097.front.sepia.ceph.com 2023-01-24T16:52:35.482 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-content-sha256:1ef0ae7bbe4ce6c99ab744fe8c27582178d69c660538ef6a4b201cf5a944e17a 2023-01-24T16:52:35.482 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-date:20230124T165151Z 2023-01-24T16:52:35.482 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.483 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5;host;x-amz-content-sha256;x-amz-date 2023-01-24T16:52:35.483 INFO:teuthology.orchestra.run.smithi097.stderr:1ef0ae7bbe4ce6c99ab744fe8c27582178d69c660538ef6a4b201cf5a944e17a 2023-01-24T16:52:35.483 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: StringToSign: 2023-01-24T16:52:35.483 INFO:teuthology.orchestra.run.smithi097.stderr:AWS4-HMAC-SHA256 2023-01-24T16:52:35.483 INFO:teuthology.orchestra.run.smithi097.stderr:20230124T165151Z 2023-01-24T16:52:35.484 INFO:teuthology.orchestra.run.smithi097.stderr:20230124/us-east-1/s3/aws4_request 2023-01-24T16:52:35.484 INFO:teuthology.orchestra.run.smithi097.stderr:e4be5d2bb49fc55bd53a2929d35c2f546718d0afcebfd41cb749c0439c5ddd88 2023-01-24T16:52:35.484 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Signature: 2023-01-24T16:52:35.484 INFO:teuthology.orchestra.run.smithi097.stderr:ae8bea300c8332fc2e497f866ed015a86c7dbb375f93cbbeffc779965616e976 2023-01-24T16:52:35.484 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutObject: calling handler 2023-01-24T16:52:35.484 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-01-24T16:52:35.485 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.awsrequest: DEBUG: Waiting for 100 Continue response. 2023-01-24T16:52:35.485 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.awsrequest: DEBUG: 100 Continue response seen, now sending request body. 2023-01-24T16:52:35.485 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: http://smithi097.front.sepia.ceph.com:80 "PUT /test-client.0-767ump700z8ujae-34/test1/a HTTP/1.1" 200 0 2023-01-24T16:52:35.485 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response headers: {'Content-Length': '0', 'ETag': '"928cae62314b95527918ee4a2447da01"', 'Accept-Ranges': 'bytes', 'x-amz-version-id': 'PW6xRDtkzuwQWjm84NZLK65jWBMjHW7', 'x-amz-expiration': 'expiry-date="Fri, 03 Feb 2023 00:00:00 GMT", rule-id="rule1"', 'x-amz-request-id': 'tx00000e6de2a19ebc73876-0063d00ca7-1089-default', 'Date': 'Tue, 24 Jan 2023 16:51:51 GMT', 'Connection': 'Keep-Alive'} 2023-01-24T16:52:35.486 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response body: 2023-01-24T16:52:35.486 INFO:teuthology.orchestra.run.smithi097.stderr:b'' 2023-01-24T16:52:35.486 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutObject: calling handler 2023-01-24T16:52:35.486 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-01-24T16:52:35.486 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutObject: calling handler > 2023-01-24T16:52:35.486 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.487 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.487 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.487 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.487 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler > 2023-01-24T16:52:35.487 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler > 2023-01-24T16:52:35.488 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.488 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.488 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.488 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.handlers: DEBUG: Adding expect 100 continue header to request. 2023-01-24T16:52:35.488 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler > 2023-01-24T16:52:35.489 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.489 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.489 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=PutObject) with params: {'url_path': '/test-client.0-767ump700z8ujae-34/test1/a', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-305.25.1.el8_4.x86_64 Botocore/1.26.10', 'Content-MD5': 'YQvR/MQ+iYleyHFitlImww==', 'Expect': '100-continue'}, 'body': <_io.BytesIO object at 0x7f26ecc1d830>, 'url': 'http://smithi097.front.sepia.ceph.com:80/test-client.0-767ump700z8ujae-34/test1/a', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': True, 'auth_type': None, 'signing': {'bucket': 'test-client.0-767ump700z8ujae-34'}}} 2023-01-24T16:52:35.489 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutObject: calling handler > 2023-01-24T16:52:35.489 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler > 2023-01-24T16:52:35.490 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler 2023-01-24T16:52:35.490 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-sign.s3.PutObject: calling handler > 2023-01-24T16:52:35.490 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-01-24T16:52:35.490 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-01-24T16:52:35.490 INFO:teuthology.orchestra.run.smithi097.stderr:PUT 2023-01-24T16:52:35.490 INFO:teuthology.orchestra.run.smithi097.stderr:/test-client.0-767ump700z8ujae-34/test1/a 2023-01-24T16:52:35.491 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.491 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5:YQvR/MQ+iYleyHFitlImww== 2023-01-24T16:52:35.491 INFO:teuthology.orchestra.run.smithi097.stderr:host:smithi097.front.sepia.ceph.com 2023-01-24T16:52:35.491 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-content-sha256:3460ebae1c45bfd069074b365281354cfdf41b82ffb05c7eedd6775446fcd3a4 2023-01-24T16:52:35.491 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-date:20230124T165151Z 2023-01-24T16:52:35.492 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.492 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5;host;x-amz-content-sha256;x-amz-date 2023-01-24T16:52:35.492 INFO:teuthology.orchestra.run.smithi097.stderr:3460ebae1c45bfd069074b365281354cfdf41b82ffb05c7eedd6775446fcd3a4 2023-01-24T16:52:35.492 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: StringToSign: 2023-01-24T16:52:35.492 INFO:teuthology.orchestra.run.smithi097.stderr:AWS4-HMAC-SHA256 2023-01-24T16:52:35.493 INFO:teuthology.orchestra.run.smithi097.stderr:20230124T165151Z 2023-01-24T16:52:35.493 INFO:teuthology.orchestra.run.smithi097.stderr:20230124/us-east-1/s3/aws4_request 2023-01-24T16:52:35.493 INFO:teuthology.orchestra.run.smithi097.stderr:285549141cc3fc9f2f215edd5d82e48fe05d5814f620bc731872b847051f1114 2023-01-24T16:52:35.493 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Signature: 2023-01-24T16:52:35.493 INFO:teuthology.orchestra.run.smithi097.stderr:9e8d712b377ad9dd806e279585f297403dc1516413dafa89988e8429b728c07a 2023-01-24T16:52:35.494 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutObject: calling handler 2023-01-24T16:52:35.494 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-01-24T16:52:35.494 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.awsrequest: DEBUG: Waiting for 100 Continue response. 2023-01-24T16:52:35.494 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.awsrequest: DEBUG: 100 Continue response seen, now sending request body. 2023-01-24T16:52:35.494 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: http://smithi097.front.sepia.ceph.com:80 "PUT /test-client.0-767ump700z8ujae-34/test1/a HTTP/1.1" 200 0 2023-01-24T16:52:35.494 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response headers: {'Content-Length': '0', 'ETag': '"610bd1fcc43e89895ec87162b65226c3"', 'Accept-Ranges': 'bytes', 'x-amz-version-id': '.6jD7hcJmAEZXvJ9jc2H1rCc-FG8qo7', 'x-amz-expiration': 'expiry-date="Fri, 03 Feb 2023 00:00:00 GMT", rule-id="rule1"', 'x-amz-request-id': 'tx00000618aa024a206a458-0063d00ca7-1089-default', 'Date': 'Tue, 24 Jan 2023 16:51:51 GMT', 'Connection': 'Keep-Alive'} 2023-01-24T16:52:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response body: 2023-01-24T16:52:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:b'' 2023-01-24T16:52:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutObject: calling handler 2023-01-24T16:52:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-01-24T16:52:35.495 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutObject: calling handler > 2023-01-24T16:52:35.496 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.496 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.496 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.496 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.496 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler > 2023-01-24T16:52:35.497 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler > 2023-01-24T16:52:35.497 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.497 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.497 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.497 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.handlers: DEBUG: Adding expect 100 continue header to request. 2023-01-24T16:52:35.498 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler > 2023-01-24T16:52:35.498 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.498 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.498 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=PutObject) with params: {'url_path': '/test-client.0-767ump700z8ujae-34/test1/b', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-305.25.1.el8_4.x86_64 Botocore/1.26.10', 'Content-MD5': 'jmibqIuS5AIMZQjopz5OwQ==', 'Expect': '100-continue'}, 'body': <_io.BytesIO object at 0x7f26ecc1d830>, 'url': 'http://smithi097.front.sepia.ceph.com:80/test-client.0-767ump700z8ujae-34/test1/b', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': True, 'auth_type': None, 'signing': {'bucket': 'test-client.0-767ump700z8ujae-34'}}} 2023-01-24T16:52:35.498 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutObject: calling handler > 2023-01-24T16:52:35.498 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler > 2023-01-24T16:52:35.499 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler 2023-01-24T16:52:35.499 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-sign.s3.PutObject: calling handler > 2023-01-24T16:52:35.499 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-01-24T16:52:35.499 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-01-24T16:52:35.499 INFO:teuthology.orchestra.run.smithi097.stderr:PUT 2023-01-24T16:52:35.500 INFO:teuthology.orchestra.run.smithi097.stderr:/test-client.0-767ump700z8ujae-34/test1/b 2023-01-24T16:52:35.500 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.500 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5:jmibqIuS5AIMZQjopz5OwQ== 2023-01-24T16:52:35.500 INFO:teuthology.orchestra.run.smithi097.stderr:host:smithi097.front.sepia.ceph.com 2023-01-24T16:52:35.500 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-content-sha256:913bea5c2f5b94d7eede7a4ddec9110d312682917cbc114eb4f729a26428d02c 2023-01-24T16:52:35.500 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-date:20230124T165151Z 2023-01-24T16:52:35.501 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.501 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5;host;x-amz-content-sha256;x-amz-date 2023-01-24T16:52:35.501 INFO:teuthology.orchestra.run.smithi097.stderr:913bea5c2f5b94d7eede7a4ddec9110d312682917cbc114eb4f729a26428d02c 2023-01-24T16:52:35.501 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: StringToSign: 2023-01-24T16:52:35.501 INFO:teuthology.orchestra.run.smithi097.stderr:AWS4-HMAC-SHA256 2023-01-24T16:52:35.502 INFO:teuthology.orchestra.run.smithi097.stderr:20230124T165151Z 2023-01-24T16:52:35.502 INFO:teuthology.orchestra.run.smithi097.stderr:20230124/us-east-1/s3/aws4_request 2023-01-24T16:52:35.502 INFO:teuthology.orchestra.run.smithi097.stderr:afbf3269940e8c020915cbe741302c90b3b246d5671b8216d8b472922c9cc852 2023-01-24T16:52:35.502 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Signature: 2023-01-24T16:52:35.502 INFO:teuthology.orchestra.run.smithi097.stderr:db23fdad525e890acb6815a3cbf1cc3207ce9db79a2eff38a3aa579c7525c851 2023-01-24T16:52:35.502 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutObject: calling handler 2023-01-24T16:52:35.503 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-01-24T16:52:35.503 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.awsrequest: DEBUG: Waiting for 100 Continue response. 2023-01-24T16:52:35.503 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.awsrequest: DEBUG: 100 Continue response seen, now sending request body. 2023-01-24T16:52:35.503 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: http://smithi097.front.sepia.ceph.com:80 "PUT /test-client.0-767ump700z8ujae-34/test1/b HTTP/1.1" 200 0 2023-01-24T16:52:35.503 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response headers: {'Content-Length': '0', 'ETag': '"8e689ba88b92e4020c6508e8a73e4ec1"', 'Accept-Ranges': 'bytes', 'x-amz-version-id': 'v8Y8l1i7sMgWazTnpiGPNUdsR.prHZt', 'x-amz-expiration': 'expiry-date="Fri, 03 Feb 2023 00:00:00 GMT", rule-id="rule1"', 'x-amz-request-id': 'tx0000038636f822ecec05c-0063d00ca7-1089-default', 'Date': 'Tue, 24 Jan 2023 16:51:51 GMT', 'Connection': 'Keep-Alive'} 2023-01-24T16:52:35.504 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response body: 2023-01-24T16:52:35.504 INFO:teuthology.orchestra.run.smithi097.stderr:b'' 2023-01-24T16:52:35.504 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutObject: calling handler 2023-01-24T16:52:35.504 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-01-24T16:52:35.504 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutObject: calling handler > 2023-01-24T16:52:35.505 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.505 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.505 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.505 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.505 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler > 2023-01-24T16:52:35.506 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler > 2023-01-24T16:52:35.506 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.506 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.506 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.506 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.handlers: DEBUG: Adding expect 100 continue header to request. 2023-01-24T16:52:35.507 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler > 2023-01-24T16:52:35.507 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.507 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.507 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=PutObject) with params: {'url_path': '/test-client.0-767ump700z8ujae-34/test1/b', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-305.25.1.el8_4.x86_64 Botocore/1.26.10', 'Content-MD5': 'koyuYjFLlVJ5GO5KJEfaAQ==', 'Expect': '100-continue'}, 'body': <_io.BytesIO object at 0x7f26ecc1dba0>, 'url': 'http://smithi097.front.sepia.ceph.com:80/test-client.0-767ump700z8ujae-34/test1/b', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': True, 'auth_type': None, 'signing': {'bucket': 'test-client.0-767ump700z8ujae-34'}}} 2023-01-24T16:52:35.507 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutObject: calling handler > 2023-01-24T16:52:35.507 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler > 2023-01-24T16:52:35.508 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler 2023-01-24T16:52:35.508 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-sign.s3.PutObject: calling handler > 2023-01-24T16:52:35.510 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-01-24T16:52:35.510 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-01-24T16:52:35.510 INFO:teuthology.orchestra.run.smithi097.stderr:PUT 2023-01-24T16:52:35.510 INFO:teuthology.orchestra.run.smithi097.stderr:/test-client.0-767ump700z8ujae-34/test1/b 2023-01-24T16:52:35.511 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.511 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5:koyuYjFLlVJ5GO5KJEfaAQ== 2023-01-24T16:52:35.511 INFO:teuthology.orchestra.run.smithi097.stderr:host:smithi097.front.sepia.ceph.com 2023-01-24T16:52:35.511 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-content-sha256:1ef0ae7bbe4ce6c99ab744fe8c27582178d69c660538ef6a4b201cf5a944e17a 2023-01-24T16:52:35.511 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-date:20230124T165151Z 2023-01-24T16:52:35.512 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.512 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5;host;x-amz-content-sha256;x-amz-date 2023-01-24T16:52:35.512 INFO:teuthology.orchestra.run.smithi097.stderr:1ef0ae7bbe4ce6c99ab744fe8c27582178d69c660538ef6a4b201cf5a944e17a 2023-01-24T16:52:35.512 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: StringToSign: 2023-01-24T16:52:35.512 INFO:teuthology.orchestra.run.smithi097.stderr:AWS4-HMAC-SHA256 2023-01-24T16:52:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:20230124T165151Z 2023-01-24T16:52:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:20230124/us-east-1/s3/aws4_request 2023-01-24T16:52:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:fcbe9a35ee6aeea473229d907d384f966c5b3ecf6b9b618c1ea9d545e23bfa2a 2023-01-24T16:52:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Signature: 2023-01-24T16:52:35.513 INFO:teuthology.orchestra.run.smithi097.stderr:5ac419f92847c5e934a7f69dc1004bec187a72cc43f9e732f3615422141c3c81 2023-01-24T16:52:35.514 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutObject: calling handler 2023-01-24T16:52:35.514 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-01-24T16:52:35.514 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.awsrequest: DEBUG: Waiting for 100 Continue response. 2023-01-24T16:52:35.514 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.awsrequest: DEBUG: 100 Continue response seen, now sending request body. 2023-01-24T16:52:35.514 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: http://smithi097.front.sepia.ceph.com:80 "PUT /test-client.0-767ump700z8ujae-34/test1/b HTTP/1.1" 200 0 2023-01-24T16:52:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response headers: {'Content-Length': '0', 'ETag': '"928cae62314b95527918ee4a2447da01"', 'Accept-Ranges': 'bytes', 'x-amz-version-id': '2QkJirSZk1qktPNTaRR.4pSMjmr5QTp', 'x-amz-expiration': 'expiry-date="Fri, 03 Feb 2023 00:00:00 GMT", rule-id="rule1"', 'x-amz-request-id': 'tx000005a46eceae3be2020-0063d00ca7-1089-default', 'Date': 'Tue, 24 Jan 2023 16:51:51 GMT', 'Connection': 'Keep-Alive'} 2023-01-24T16:52:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response body: 2023-01-24T16:52:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:b'' 2023-01-24T16:52:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutObject: calling handler 2023-01-24T16:52:35.515 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-01-24T16:52:35.516 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutObject: calling handler > 2023-01-24T16:52:35.516 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.516 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.516 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.516 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.517 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler > 2023-01-24T16:52:35.517 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler > 2023-01-24T16:52:35.517 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutObject: calling handler 2023-01-24T16:52:35.517 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.517 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.517 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.handlers: DEBUG: Adding expect 100 continue header to request. 2023-01-24T16:52:35.518 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler > 2023-01-24T16:52:35.518 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.518 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutObject: calling handler 2023-01-24T16:52:35.518 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=PutObject) with params: {'url_path': '/test-client.0-767ump700z8ujae-34/test1/b', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-305.25.1.el8_4.x86_64 Botocore/1.26.10', 'Content-MD5': 'YQvR/MQ+iYleyHFitlImww==', 'Expect': '100-continue'}, 'body': <_io.BytesIO object at 0x7f26ecc1dba0>, 'url': 'http://smithi097.front.sepia.ceph.com:80/test-client.0-767ump700z8ujae-34/test1/b', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': True, 'auth_type': None, 'signing': {'bucket': 'test-client.0-767ump700z8ujae-34'}}} 2023-01-24T16:52:35.518 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutObject: calling handler > 2023-01-24T16:52:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler > 2023-01-24T16:52:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler 2023-01-24T16:52:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-sign.s3.PutObject: calling handler > 2023-01-24T16:52:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-01-24T16:52:35.519 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-01-24T16:52:35.520 INFO:teuthology.orchestra.run.smithi097.stderr:PUT 2023-01-24T16:52:35.520 INFO:teuthology.orchestra.run.smithi097.stderr:/test-client.0-767ump700z8ujae-34/test1/b 2023-01-24T16:52:35.520 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.520 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5:YQvR/MQ+iYleyHFitlImww== 2023-01-24T16:52:35.520 INFO:teuthology.orchestra.run.smithi097.stderr:host:smithi097.front.sepia.ceph.com 2023-01-24T16:52:35.521 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-content-sha256:3460ebae1c45bfd069074b365281354cfdf41b82ffb05c7eedd6775446fcd3a4 2023-01-24T16:52:35.521 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-date:20230124T165151Z 2023-01-24T16:52:35.521 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.521 INFO:teuthology.orchestra.run.smithi097.stderr:content-md5;host;x-amz-content-sha256;x-amz-date 2023-01-24T16:52:35.521 INFO:teuthology.orchestra.run.smithi097.stderr:3460ebae1c45bfd069074b365281354cfdf41b82ffb05c7eedd6775446fcd3a4 2023-01-24T16:52:35.522 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: StringToSign: 2023-01-24T16:52:35.522 INFO:teuthology.orchestra.run.smithi097.stderr:AWS4-HMAC-SHA256 2023-01-24T16:52:35.522 INFO:teuthology.orchestra.run.smithi097.stderr:20230124T165151Z 2023-01-24T16:52:35.523 INFO:teuthology.orchestra.run.smithi097.stderr:20230124/us-east-1/s3/aws4_request 2023-01-24T16:52:35.523 INFO:teuthology.orchestra.run.smithi097.stderr:8608575d498987b512d71701fc1f3d2a6e992486f8f62757ccd2752fc7ef131a 2023-01-24T16:52:35.523 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Signature: 2023-01-24T16:52:35.523 INFO:teuthology.orchestra.run.smithi097.stderr:2d75acbfd56c0122abc27b4610e4770d3af4e59c04931e622f41117cdcd94230 2023-01-24T16:52:35.523 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutObject: calling handler 2023-01-24T16:52:35.524 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-01-24T16:52:35.524 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.awsrequest: DEBUG: Waiting for 100 Continue response. 2023-01-24T16:52:35.524 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.awsrequest: DEBUG: 100 Continue response seen, now sending request body. 2023-01-24T16:52:35.524 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: http://smithi097.front.sepia.ceph.com:80 "PUT /test-client.0-767ump700z8ujae-34/test1/b HTTP/1.1" 200 0 2023-01-24T16:52:35.525 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response headers: {'Content-Length': '0', 'ETag': '"610bd1fcc43e89895ec87162b65226c3"', 'Accept-Ranges': 'bytes', 'x-amz-version-id': 'RnFZ050-t8xTyyRSApXR1GEu4ujncS5', 'x-amz-expiration': 'expiry-date="Fri, 03 Feb 2023 00:00:00 GMT", rule-id="rule1"', 'x-amz-request-id': 'tx0000052b8483a363c2bdb-0063d00ca7-1089-default', 'Date': 'Tue, 24 Jan 2023 16:51:51 GMT', 'Connection': 'Keep-Alive'} 2023-01-24T16:52:35.525 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response body: 2023-01-24T16:52:35.525 INFO:teuthology.orchestra.run.smithi097.stderr:b'' 2023-01-24T16:52:35.525 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutObject: calling handler 2023-01-24T16:52:35.525 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-01-24T16:52:35.526 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutObject: calling handler > 2023-01-24T16:52:35.526 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.526 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.526 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.526 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.526 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.527 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.527 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.527 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.527 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.527 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=ListObjectVersions) with params: {'url_path': '/test-client.0-767ump700z8ujae-34?versions', 'query_string': {'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-305.25.1.el8_4.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi097.front.sepia.ceph.com:80/test-client.0-767ump700z8ujae-34?versions&encoding-type=url', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'encoding_type_auto_set': True, 'signing': {'bucket': 'test-client.0-767ump700z8ujae-34'}}} 2023-01-24T16:52:35.528 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.528 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.528 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.528 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-sign.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.529 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-01-24T16:52:35.529 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-01-24T16:52:35.529 INFO:teuthology.orchestra.run.smithi097.stderr:GET 2023-01-24T16:52:35.529 INFO:teuthology.orchestra.run.smithi097.stderr:/test-client.0-767ump700z8ujae-34 2023-01-24T16:52:35.529 INFO:teuthology.orchestra.run.smithi097.stderr:encoding-type=url&versions= 2023-01-24T16:52:35.529 INFO:teuthology.orchestra.run.smithi097.stderr:host:smithi097.front.sepia.ceph.com 2023-01-24T16:52:35.530 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-01-24T16:52:35.530 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-date:20230124T165151Z 2023-01-24T16:52:35.530 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.530 INFO:teuthology.orchestra.run.smithi097.stderr:host;x-amz-content-sha256;x-amz-date 2023-01-24T16:52:35.530 INFO:teuthology.orchestra.run.smithi097.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-01-24T16:52:35.531 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: StringToSign: 2023-01-24T16:52:35.531 INFO:teuthology.orchestra.run.smithi097.stderr:AWS4-HMAC-SHA256 2023-01-24T16:52:35.531 INFO:teuthology.orchestra.run.smithi097.stderr:20230124T165151Z 2023-01-24T16:52:35.531 INFO:teuthology.orchestra.run.smithi097.stderr:20230124/us-east-1/s3/aws4_request 2023-01-24T16:52:35.531 INFO:teuthology.orchestra.run.smithi097.stderr:ccd5953e6542b06dc741d5c3e3fdcf3f19eee3e841982ceb42da099c9ac468c1 2023-01-24T16:52:35.532 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Signature: 2023-01-24T16:52:35.532 INFO:teuthology.orchestra.run.smithi097.stderr:ca3b1d426067bfe586330fd86d299342ed0b0d0ebcaff2e912066a5200b0ac4b 2023-01-24T16:52:35.532 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.532 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-01-24T16:52:35.532 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: http://smithi097.front.sepia.ceph.com:80 "GET /test-client.0-767ump700z8ujae-34?versions&encoding-type=url HTTP/1.1" 200 None 2023-01-24T16:52:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx000001ff867c55ae87208-0063d00ca7-1089-default', 'Content-Type': 'application/xml', 'Date': 'Tue, 24 Jan 2023 16:51:51 GMT', 'Connection': 'Keep-Alive'} 2023-01-24T16:52:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response body: 2023-01-24T16:52:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:b'urltest-client.0-767ump700z8ujae-341000falsetest1%2Fa.6jD7hcJmAEZXvJ9jc2H1rCc-FG8qo7true2023-01-24T16:51:51.868Z"610bd1fcc43e89895ec87162b65226c3"9STANDARDfoo.client.0Mr. foo.client.0Normaltest1%2FaPW6xRDtkzuwQWjm84NZLK65jWBMjHW7false2023-01-24T16:51:51.857Z"928cae62314b95527918ee4a2447da01"9STANDARDfoo.client.0Mr. foo.client.0Normaltest1%2FaBFAGISa0bubO4Xh-r1MB2nv8usJk24zfalse2023-01-24T16:51:51.847Z"8e689ba88b92e4020c6508e8a73e4ec1"9STANDARDfoo.client.0Mr. foo.client.0Normaltest1%2FbRnFZ050-t8xTyyRSApXR1GEu4ujncS5true2023-01-24T16:51:51.905Z"610bd1fcc43e89895ec87162b65226c3"9STANDARDfoo.client.0Mr. foo.client.0Normaltest1%2Fb2QkJirSZk1qktPNTaRR.4pSMjmr5QTpfalse2023-01-24T16:51:51.895Z"928cae62314b95527918ee4a2447da01"9STANDARDfoo.client.0Mr. foo.client.0Normaltest1%2Fbv8Y8l1i7sMgWazTnpiGPNUdsR.prHZtfalse2023-01-24T16:51:51.882Z"8e689ba88b92e4020c6508e8a73e4ec1"9STANDARDfoo.client.0Mr. foo.client.0Normal' 2023-01-24T16:52:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.533 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-01-24T16:52:35.534 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.534 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event after-call.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.534 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.534 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.535 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.535 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.535 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.535 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.535 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.536 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.536 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-call.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.536 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=ListObjectVersions) with params: {'url_path': '/test-client.0-767ump700z8ujae-34?versions', 'query_string': {'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-305.25.1.el8_4.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi097.front.sepia.ceph.com:80/test-client.0-767ump700z8ujae-34?versions&encoding-type=url', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'encoding_type_auto_set': True, 'signing': {'bucket': 'test-client.0-767ump700z8ujae-34'}}} 2023-01-24T16:52:35.536 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.536 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.537 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.537 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event before-sign.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.537 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-01-24T16:52:35.537 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-01-24T16:52:35.537 INFO:teuthology.orchestra.run.smithi097.stderr:GET 2023-01-24T16:52:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:/test-client.0-767ump700z8ujae-34 2023-01-24T16:52:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:encoding-type=url&versions= 2023-01-24T16:52:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:host:smithi097.front.sepia.ceph.com 2023-01-24T16:52:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-01-24T16:52:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:x-amz-date:20230124T165231Z 2023-01-24T16:52:35.539 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:host;x-amz-content-sha256;x-amz-date 2023-01-24T16:52:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-01-24T16:52:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: StringToSign: 2023-01-24T16:52:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:AWS4-HMAC-SHA256 2023-01-24T16:52:35.540 INFO:teuthology.orchestra.run.smithi097.stderr:20230124T165231Z 2023-01-24T16:52:35.540 INFO:teuthology.orchestra.run.smithi097.stderr:20230124/us-east-1/s3/aws4_request 2023-01-24T16:52:35.540 INFO:teuthology.orchestra.run.smithi097.stderr:0b1b73ed5ca324ce53f254988d9ea5d783594600ddc9f8d09514e6b259f8ea77 2023-01-24T16:52:35.540 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.auth: DEBUG: Signature: 2023-01-24T16:52:35.540 INFO:teuthology.orchestra.run.smithi097.stderr:760bd1471d5abaef91f6d8328bd536aea6b55ac0515b80e30c012ae2cf1e11f2 2023-01-24T16:52:35.540 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event request-created.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.541 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-01-24T16:52:35.541 INFO:teuthology.orchestra.run.smithi097.stderr:urllib3.connectionpool: DEBUG: http://smithi097.front.sepia.ceph.com:80 "GET /test-client.0-767ump700z8ujae-34?versions&encoding-type=url HTTP/1.1" 200 None 2023-01-24T16:52:35.541 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response headers: {'Transfer-Encoding': 'chunked', 'x-amz-request-id': 'tx00000c8eee153d25e3b3a-0063d00ccf-1089-default', 'Content-Type': 'application/xml', 'Date': 'Tue, 24 Jan 2023 16:52:31 GMT', 'Connection': 'Keep-Alive'} 2023-01-24T16:52:35.541 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.parsers: DEBUG: Response body: 2023-01-24T16:52:35.542 INFO:teuthology.orchestra.run.smithi097.stderr:b'urltest-client.0-767ump700z8ujae-341000falsetest1%2Fa.6jD7hcJmAEZXvJ9jc2H1rCc-FG8qo7true2023-01-24T16:51:51.868Z"610bd1fcc43e89895ec87162b65226c3"9STANDARDfoo.client.0Mr. foo.client.0Normaltest1%2FaPW6xRDtkzuwQWjm84NZLK65jWBMjHW7false2023-01-24T16:51:51.857Z"928cae62314b95527918ee4a2447da01"9STANDARDfoo.client.0Mr. foo.client.0Normaltest1%2FaBFAGISa0bubO4Xh-r1MB2nv8usJk24zfalse2023-01-24T16:51:51.847Z"8e689ba88b92e4020c6508e8a73e4ec1"9STANDARDfoo.client.0Mr. foo.client.0Normaltest1%2FbRnFZ050-t8xTyyRSApXR1GEu4ujncS5true2023-01-24T16:51:51.905Z"610bd1fcc43e89895ec87162b65226c3"9STANDARDfoo.client.0Mr. foo.client.0Normaltest1%2Fb2QkJirSZk1qktPNTaRR.4pSMjmr5QTpfalse2023-01-24T16:51:51.895Z"928cae62314b95527918ee4a2447da01"9STANDARDfoo.client.0Mr. foo.client.0Normaltest1%2Fbv8Y8l1i7sMgWazTnpiGPNUdsR.prHZtfalse2023-01-24T16:51:51.882Z"8e689ba88b92e4020c6508e8a73e4ec1"9STANDARDfoo.client.0Mr. foo.client.0Normal' 2023-01-24T16:52:35.542 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.542 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-01-24T16:52:35.542 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.ListObjectVersions: calling handler > 2023-01-24T16:52:35.542 INFO:teuthology.orchestra.run.smithi097.stderr:botocore.hooks: DEBUG: Event after-call.s3.ListObjectVersions: calling handler 2023-01-24T16:52:35.543 INFO:teuthology.orchestra.run.smithi097.stderr:--------------------- >> end captured logging << --------------------- 2023-01-24T16:52:35.543 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.543 INFO:teuthology.orchestra.run.smithi097.stderr:---------------------------------------------------------------------- 2023-01-24T16:52:35.543 INFO:teuthology.orchestra.run.smithi097.stderr:Ran 38 tests in 798.839s 2023-01-24T16:52:35.543 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.543 INFO:teuthology.orchestra.run.smithi097.stderr:FAILED (SKIP=4, failures=1) 2023-01-24T16:52:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T16:52:35.545 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa/tasks/s3tests.py", line 456, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi097 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a lifecycle' 2023-01-24T16:52:35.546 INFO:tasks.s3tests:Cleaning up boto... 2023-01-24T16:52:35.547 DEBUG:teuthology.orchestra.run.smithi097:> rm /home/ubuntu/cephtest/boto.cfg 2023-01-24T16:52:35.563 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2023-01-24T16:52:35.642 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:52:35.642 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:52:35.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.647+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.649 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.648+0000 7f9585966680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.648+0000 7f9585966680 20 realm 2023-01-24T16:52:35.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.648+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.648+0000 7f9585966680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.648+0000 7f9585966680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T16:52:35.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.648+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.651 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.648+0000 7f9585966680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.651 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.648+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.651 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.649+0000 7f9585966680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:35.651 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.649+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.651 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.649+0000 7f9585966680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:52:35.652 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.649+0000 7f9585966680 20 searching for the correct realm 2023-01-24T16:52:35.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.656+0000 7f9585966680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:52:35.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.656+0000 7f9585966680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:52:35.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.656+0000 7f9585966680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:52:35.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.656+0000 7f9585966680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:52:35.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.656+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.656+0000 7f9585966680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.656+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.657+0000 7f9585966680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:35.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.657+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.657+0000 7f9585966680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:52:35.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.657+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.657+0000 7f9585966680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.657+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.657+0000 7f9585966680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.657+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.657+0000 7f9585966680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:35.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.663+0000 7f9585966680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:52:35.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.663+0000 7f9585966680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:52:35.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.663+0000 7f9585966680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:52:35.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.663+0000 7f9585966680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:52:35.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.663+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.663+0000 7f9585966680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:35.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:52:35.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.669 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:35.669 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.669 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:52:35.669 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.669 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 20 zone default found 2023-01-24T16:52:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 4 Realm: () 2023-01-24T16:52:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 4 ZoneGroup: default (2097a68d-863f-4181-8202-77da07845417) 2023-01-24T16:52:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 4 Zone: default (e9b1a570-bc22-4f42-b7c6-96dbb40f320d) 2023-01-24T16:52:35.670 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T16:52:35.671 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 20 zonegroup default 2023-01-24T16:52:35.671 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.664+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.671 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.665+0000 7f9585966680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.671 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.665+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.671 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.665+0000 7f9585966680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.672 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.665+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.672 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.665+0000 7f9585966680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.672 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.665+0000 7f9585966680 20 started sync module instance, tier type = 2023-01-24T16:52:35.672 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.665+0000 7f9585966680 20 started zone id=e9b1a570-bc22-4f42-b7c6-96dbb40f320d (name=default) with tier type = 2023-01-24T16:52:35.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.695+0000 7f9585966680 20 add_watcher() i=0 2023-01-24T16:52:35.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.695+0000 7f9585966680 20 add_watcher() i=1 2023-01-24T16:52:35.697 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.695+0000 7f9585966680 20 add_watcher() i=2 2023-01-24T16:52:35.697 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.695+0000 7f9585966680 20 add_watcher() i=3 2023-01-24T16:52:35.697 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.695+0000 7f9585966680 20 add_watcher() i=4 2023-01-24T16:52:35.697 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.695+0000 7f9585966680 20 add_watcher() i=5 2023-01-24T16:52:35.697 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.695+0000 7f9585966680 20 add_watcher() i=6 2023-01-24T16:52:35.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.696+0000 7f9585966680 20 add_watcher() i=7 2023-01-24T16:52:35.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.696+0000 7f9585966680 2 all 8 watchers are set, enabling cache 2023-01-24T16:52:35.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.698+0000 7f9549fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:52:35.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.698+0000 7f9585966680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:52:35.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.698+0000 7f9585966680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:52:35.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.751+0000 7f94c0ff9700 20 reqs_thread_entry: start 2023-01-24T16:52:35.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.751+0000 7f94b67fc700 20 reqs_thread_entry: start 2023-01-24T16:52:35.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.751+0000 7f9585966680 20 init_complete bucket index max shards: 11 2023-01-24T16:52:35.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.751+0000 7f9585966680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T16:52:35.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.751+0000 7f94b4ff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 16:53:06 2023 2023-01-24T16:52:35.755 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.751+0000 7f949bfff700 20 reqs_thread_entry: start 2023-01-24T16:52:35.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.751+0000 7f9585966680 20 get_system_obj_state: rctx=0x7ffdf597a0d8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x5634f06f0940 s->prefetch_data=0 2023-01-24T16:52:35.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.751+0000 7f9585966680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-01-24T16:52:35.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-01-24T16:52:35.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T16:52:35.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T16:52:35.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T16:52:35.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 20 Read xattr: user.rgw.idtag 2023-01-24T16:52:35.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T16:52:35.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 20 rados_obj.operate() r=0 bl.length=469 2023-01-24T16:52:35.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-01-24T16:52:35.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T16:52:35.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T16:52:35.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 10 removing key index: BNAVUMGDINJPSXUTEDAY 2023-01-24T16:52:35.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.752+0000 7f9585966680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.keys:BNAVUMGDINJPSXUTEDAY, ofs0, ns] 2023-01-24T16:52:35.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.753+0000 7f95727fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659214 cookie 94785373720768 notifier 4282 bl.length()=170 2023-01-24T16:52:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.754+0000 7f9585966680 10 removing email index: foo.client.0+test@test.test 2023-01-24T16:52:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.754+0000 7f9585966680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-01-24T16:52:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.754+0000 7f9571ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659159 cookie 94785373709280 notifier 4282 bl.length()=178 2023-01-24T16:52:35.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.755+0000 7f9585966680 10 removing user buckets index 2023-01-24T16:52:35.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.755+0000 7f9585966680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2023-01-24T16:52:35.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.756+0000 7f9571ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659160 cookie 94785373700400 notifier 4282 bl.length()=169 2023-01-24T16:52:35.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.757+0000 7f9585966680 10 removing user index: foo.client.0 2023-01-24T16:52:35.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.757+0000 7f9585966680 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2023-01-24T16:52:35.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.757+0000 7f9585966680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-01-24T16:52:35.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.757+0000 7f95727fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659215 cookie 94785373696800 notifier 4282 bl.length()=161 2023-01-24T16:52:35.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.759+0000 7f9585966680 20 remove_watcher() i=0 2023-01-24T16:52:35.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.759+0000 7f9585966680 2 removed watcher, disabling cache 2023-01-24T16:52:35.763 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.762+0000 7f9585966680 20 remove_watcher() i=1 2023-01-24T16:52:35.766 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.765+0000 7f9585966680 20 remove_watcher() i=2 2023-01-24T16:52:35.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.768+0000 7f9585966680 20 remove_watcher() i=3 2023-01-24T16:52:35.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.770+0000 7f9585966680 20 remove_watcher() i=4 2023-01-24T16:52:35.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.773+0000 7f9585966680 20 remove_watcher() i=5 2023-01-24T16:52:35.775 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.774+0000 7f9585966680 20 remove_watcher() i=6 2023-01-24T16:52:35.776 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.776+0000 7f9585966680 20 remove_watcher() i=7 2023-01-24T16:52:35.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.777+0000 7f9549fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:52:35.783 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2023-01-24T16:52:35.862 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:52:35.862 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:52:35.867 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.867+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.868+0000 7fe77dcdd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.868+0000 7fe77dcdd680 20 realm 2023-01-24T16:52:35.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.868+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.868+0000 7fe77dcdd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.868+0000 7fe77dcdd680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T16:52:35.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.868+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.868+0000 7fe77dcdd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.868+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.871 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.868+0000 7fe77dcdd680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:35.871 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.868+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.871 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.868+0000 7fe77dcdd680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:52:35.871 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.868+0000 7fe77dcdd680 20 searching for the correct realm 2023-01-24T16:52:35.877 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.875+0000 7fe77dcdd680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:52:35.877 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.875+0000 7fe77dcdd680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:52:35.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.875+0000 7fe77dcdd680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:52:35.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.875+0000 7fe77dcdd680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:52:35.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.875+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.876+0000 7fe77dcdd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.876+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.876+0000 7fe77dcdd680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:35.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.876+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.876+0000 7fe77dcdd680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:52:35.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.876+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.880 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.876+0000 7fe77dcdd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.880 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.876+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.880 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.876+0000 7fe77dcdd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.880 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.876+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.880 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.877+0000 7fe77dcdd680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:35.884 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.883+0000 7fe77dcdd680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:52:35.885 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.883+0000 7fe77dcdd680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:52:35.885 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.883+0000 7fe77dcdd680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:52:35.885 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.883+0000 7fe77dcdd680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:52:35.885 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.883+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.885 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.884+0000 7fe77dcdd680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:35.886 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.884+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.886 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.884+0000 7fe77dcdd680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:52:35.886 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.884+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.886 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.884+0000 7fe77dcdd680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:35.886 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.884+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.884+0000 7fe77dcdd680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:52:35.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.884+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.889 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 20 zone default found 2023-01-24T16:52:35.889 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 4 Realm: () 2023-01-24T16:52:35.889 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 4 ZoneGroup: default (2097a68d-863f-4181-8202-77da07845417) 2023-01-24T16:52:35.889 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 4 Zone: default (e9b1a570-bc22-4f42-b7c6-96dbb40f320d) 2023-01-24T16:52:35.889 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T16:52:35.890 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 20 zonegroup default 2023-01-24T16:52:35.890 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.890 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.890 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.890 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.891 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.891 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:35.891 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 20 started sync module instance, tier type = 2023-01-24T16:52:35.891 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.885+0000 7fe77dcdd680 20 started zone id=e9b1a570-bc22-4f42-b7c6-96dbb40f320d (name=default) with tier type = 2023-01-24T16:52:35.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.914+0000 7fe77dcdd680 20 add_watcher() i=0 2023-01-24T16:52:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.914+0000 7fe77dcdd680 20 add_watcher() i=1 2023-01-24T16:52:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.914+0000 7fe77dcdd680 20 add_watcher() i=2 2023-01-24T16:52:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.914+0000 7fe77dcdd680 20 add_watcher() i=3 2023-01-24T16:52:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.914+0000 7fe77dcdd680 20 add_watcher() i=4 2023-01-24T16:52:35.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.914+0000 7fe77dcdd680 20 add_watcher() i=5 2023-01-24T16:52:35.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.914+0000 7fe77dcdd680 20 add_watcher() i=6 2023-01-24T16:52:35.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.914+0000 7fe77dcdd680 20 add_watcher() i=7 2023-01-24T16:52:35.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.914+0000 7fe77dcdd680 2 all 8 watchers are set, enabling cache 2023-01-24T16:52:35.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.916+0000 7fe741fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:52:35.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.916+0000 7fe77dcdd680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:52:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.916+0000 7fe77dcdd680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:52:35.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.966+0000 7fe6b8ff9700 20 reqs_thread_entry: start 2023-01-24T16:52:35.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.966+0000 7fe6ae7fc700 20 reqs_thread_entry: start 2023-01-24T16:52:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.966+0000 7fe77dcdd680 20 init_complete bucket index max shards: 11 2023-01-24T16:52:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.966+0000 7fe77dcdd680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T16:52:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.966+0000 7fe6acff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 16:53:06 2023 2023-01-24T16:52:35.969 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:35.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.966+0000 7fe693fff700 20 reqs_thread_entry: start 2023-01-24T16:52:35.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.966+0000 7fe77dcdd680 20 get_system_obj_state: rctx=0x7fff5f3b6368 obj=default.rgw.meta:users.uid:bar.client.0 state=0x559ebb7fd940 s->prefetch_data=0 2023-01-24T16:52:35.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.966+0000 7fe77dcdd680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-01-24T16:52:35.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-01-24T16:52:35.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T16:52:35.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T16:52:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T16:52:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 20 Read xattr: user.rgw.idtag 2023-01-24T16:52:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T16:52:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 20 rados->read ofs=0 len=0 2023-01-24T16:52:35.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 20 rados_obj.operate() r=0 bl.length=469 2023-01-24T16:52:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-01-24T16:52:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T16:52:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T16:52:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 10 removing key index: HSDBKVCSJGJVYDFCJHPD 2023-01-24T16:52:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe77dcdd680 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.keys:HSDBKVCSJGJVYDFCJHPD, ofs0, ns] 2023-01-24T16:52:35.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.967+0000 7fe76a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659286 cookie 94140240517680 notifier 4288 bl.length()=170 2023-01-24T16:52:35.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.969+0000 7fe77dcdd680 10 removing email index: bar.client.0+test@test.test 2023-01-24T16:52:35.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.969+0000 7fe77dcdd680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-01-24T16:52:35.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.969+0000 7fe76a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659161 cookie 94140240515296 notifier 4288 bl.length()=178 2023-01-24T16:52:35.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.971+0000 7fe77dcdd680 10 removing user buckets index 2023-01-24T16:52:35.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.971+0000 7fe77dcdd680 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2023-01-24T16:52:35.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.972+0000 7fe769ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659287 cookie 94140240512576 notifier 4288 bl.length()=169 2023-01-24T16:52:35.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.974+0000 7fe77dcdd680 10 removing user index: bar.client.0 2023-01-24T16:52:35.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.974+0000 7fe77dcdd680 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2023-01-24T16:52:35.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.974+0000 7fe77dcdd680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-01-24T16:52:35.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.974+0000 7fe769ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659162 cookie 94140240515296 notifier 4288 bl.length()=161 2023-01-24T16:52:35.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.977+0000 7fe77dcdd680 20 remove_watcher() i=0 2023-01-24T16:52:35.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.977+0000 7fe77dcdd680 2 removed watcher, disabling cache 2023-01-24T16:52:35.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.979+0000 7fe77dcdd680 20 remove_watcher() i=1 2023-01-24T16:52:35.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.982+0000 7fe77dcdd680 20 remove_watcher() i=2 2023-01-24T16:52:35.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.985+0000 7fe77dcdd680 20 remove_watcher() i=3 2023-01-24T16:52:35.988 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.987+0000 7fe77dcdd680 20 remove_watcher() i=4 2023-01-24T16:52:35.990 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.990+0000 7fe77dcdd680 20 remove_watcher() i=5 2023-01-24T16:52:35.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.993+0000 7fe77dcdd680 20 remove_watcher() i=6 2023-01-24T16:52:35.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.994+0000 7fe77dcdd680 20 remove_watcher() i=7 2023-01-24T16:52:35.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:35.994+0000 7fe741fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:52:36.000 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2023-01-24T16:52:36.079 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:52:36.080 INFO:teuthology.orchestra.run.smithi097.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:52:36.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.084+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.085+0000 7fcdee56a680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:36.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.085+0000 7fcdee56a680 20 realm 2023-01-24T16:52:36.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.085+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.085+0000 7fcdee56a680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:36.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.085+0000 7fcdee56a680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T16:52:36.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.085+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.085+0000 7fcdee56a680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:36.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.085+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.085+0000 7fcdee56a680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:36.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.085+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.086+0000 7fcdee56a680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:52:36.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.086+0000 7fcdee56a680 20 searching for the correct realm 2023-01-24T16:52:36.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:52:36.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:52:36.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:52:36.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:52:36.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:36.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:36.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:52:36.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:36.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.093+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.094+0000 7fcdee56a680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:36.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.094+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.094+0000 7fcdee56a680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:36.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.100+0000 7fcdee56a680 20 RGWRados::pool_iterate: got zonegroup_info.2097a68d-863f-4181-8202-77da07845417 2023-01-24T16:52:36.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.100+0000 7fcdee56a680 20 RGWRados::pool_iterate: got zone_info.e9b1a570-bc22-4f42-b7c6-96dbb40f320d 2023-01-24T16:52:36.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.100+0000 7fcdee56a680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T16:52:36.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.100+0000 7fcdee56a680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T16:52:36.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.100+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.100+0000 7fcdee56a680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:36.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.100+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.100+0000 7fcdee56a680 20 rados_obj.operate() r=0 bl.length=1003 2023-01-24T16:52:36.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.100+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T16:52:36.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 20 rados_obj.operate() r=0 bl.length=396 2023-01-24T16:52:36.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:36.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 20 zone default found 2023-01-24T16:52:36.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 4 Realm: () 2023-01-24T16:52:36.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 4 ZoneGroup: default (2097a68d-863f-4181-8202-77da07845417) 2023-01-24T16:52:36.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 4 Zone: default (e9b1a570-bc22-4f42-b7c6-96dbb40f320d) 2023-01-24T16:52:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T16:52:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 20 zonegroup default 2023-01-24T16:52:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.101+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.102+0000 7fcdee56a680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.102+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.102+0000 7fcdee56a680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T16:52:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.102+0000 7fcdee56a680 20 started sync module instance, tier type = 2023-01-24T16:52:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.102+0000 7fcdee56a680 20 started zone id=e9b1a570-bc22-4f42-b7c6-96dbb40f320d (name=default) with tier type = 2023-01-24T16:52:36.125 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.124+0000 7fcdee56a680 20 add_watcher() i=0 2023-01-24T16:52:36.125 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.124+0000 7fcdee56a680 20 add_watcher() i=1 2023-01-24T16:52:36.126 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.124+0000 7fcdee56a680 20 add_watcher() i=2 2023-01-24T16:52:36.126 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.124+0000 7fcdee56a680 20 add_watcher() i=3 2023-01-24T16:52:36.126 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.124+0000 7fcdee56a680 20 add_watcher() i=4 2023-01-24T16:52:36.126 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.124+0000 7fcdee56a680 20 add_watcher() i=5 2023-01-24T16:52:36.126 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.124+0000 7fcdee56a680 20 add_watcher() i=6 2023-01-24T16:52:36.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.126+0000 7fcdee56a680 20 add_watcher() i=7 2023-01-24T16:52:36.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.126+0000 7fcdee56a680 2 all 8 watchers are set, enabling cache 2023-01-24T16:52:36.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.128+0000 7fcdb27e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:52:36.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.128+0000 7fcdee56a680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T16:52:36.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.128+0000 7fcdee56a680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T16:52:36.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.191+0000 7fcd217fa700 20 reqs_thread_entry: start 2023-01-24T16:52:36.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.191+0000 7fcd1affd700 20 reqs_thread_entry: start 2023-01-24T16:52:36.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.191+0000 7fcdee56a680 20 init_complete bucket index max shards: 11 2023-01-24T16:52:36.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.192+0000 7fcdee56a680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T16:52:36.195 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.192+0000 7fcd197fa700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 16:53:06 2023 2023-01-24T16:52:36.195 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-01-24T16:52:36.195 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.192+0000 7fcd18ff9700 20 reqs_thread_entry: start 2023-01-24T16:52:36.195 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.192+0000 7fcdee56a680 20 get_system_obj_state: rctx=0x7ffc13d83878 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55e8b9b96940 s->prefetch_data=0 2023-01-24T16:52:36.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.192+0000 7fcdee56a680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-01-24T16:52:36.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.192+0000 7fcdee56a680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-01-24T16:52:36.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.192+0000 7fcdee56a680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T16:52:36.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.192+0000 7fcdee56a680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T16:52:36.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.192+0000 7fcdee56a680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T16:52:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.192+0000 7fcdee56a680 20 Read xattr: user.rgw.idtag 2023-01-24T16:52:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.192+0000 7fcdee56a680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T16:52:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.192+0000 7fcdee56a680 20 rados->read ofs=0 len=0 2023-01-24T16:52:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.193+0000 7fcdee56a680 20 rados_obj.operate() r=0 bl.length=528 2023-01-24T16:52:36.197 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.193+0000 7fcdee56a680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-01-24T16:52:36.198 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.193+0000 7fcdee56a680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T16:52:36.198 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.193+0000 7fcdee56a680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T16:52:36.198 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.193+0000 7fcdee56a680 10 removing key index: TFMNMIQKIHDFPLQXWVRM 2023-01-24T16:52:36.198 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.193+0000 7fcdee56a680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:TFMNMIQKIHDFPLQXWVRM, ofs0, ns] 2023-01-24T16:52:36.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.193+0000 7fcdd2ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659163 cookie 94458038314016 notifier 4294 bl.length()=170 2023-01-24T16:52:36.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.195+0000 7fcdee56a680 10 removing email index: testx$tenanteduser.client.0+test@test.test 2023-01-24T16:52:36.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.195+0000 7fcdee56a680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-01-24T16:52:36.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.195+0000 7fcdd2ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659216 cookie 94458038333488 notifier 4294 bl.length()=193 2023-01-24T16:52:36.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.196+0000 7fcdee56a680 10 removing user buckets index 2023-01-24T16:52:36.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.196+0000 7fcdee56a680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2023-01-24T16:52:36.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.197+0000 7fcdd27fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659164 cookie 94458038327680 notifier 4294 bl.length()=184 2023-01-24T16:52:36.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.198+0000 7fcdee56a680 10 removing user index: testx$tenanteduser.client.0 2023-01-24T16:52:36.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.198+0000 7fcdee56a680 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2023-01-24T16:52:36.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.198+0000 7fcdee56a680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-01-24T16:52:36.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.198+0000 7fcdd27fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659217 cookie 94458038323536 notifier 4294 bl.length()=176 2023-01-24T16:52:36.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.201+0000 7fcdee56a680 20 remove_watcher() i=0 2023-01-24T16:52:36.203 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.201+0000 7fcdee56a680 2 removed watcher, disabling cache 2023-01-24T16:52:36.203 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.202+0000 7fcdee56a680 20 remove_watcher() i=1 2023-01-24T16:52:36.203 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.203+0000 7fcdee56a680 20 remove_watcher() i=2 2023-01-24T16:52:36.204 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.204+0000 7fcdee56a680 20 remove_watcher() i=3 2023-01-24T16:52:36.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.205+0000 7fcdee56a680 20 remove_watcher() i=4 2023-01-24T16:52:36.206 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.206+0000 7fcdee56a680 20 remove_watcher() i=5 2023-01-24T16:52:36.208 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.208+0000 7fcdee56a680 20 remove_watcher() i=6 2023-01-24T16:52:36.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.208+0000 7fcdee56a680 20 remove_watcher() i=7 2023-01-24T16:52:36.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-01-24T16:52:36.209+0000 7fcdb27e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T16:52:36.215 INFO:tasks.s3tests:Removing s3-tests... 2023-01-24T16:52:36.216 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf /home/ubuntu/cephtest/s3-tests 2023-01-24T16:52:36.378 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa/tasks/s3tests.py", line 654, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/contextutil.py", line 55, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa/tasks/s3tests.py", line 59, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa/tasks/s3tests.py", line 456, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi097 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a lifecycle' 2023-01-24T16:52:36.446 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=82e5f46a5c8a43348e2d4e8652604f37 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa/tasks/s3tests.py", line 654, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/contextutil.py", line 55, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa/tasks/s3tests.py", line 59, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa/tasks/s3tests.py", line 456, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi097 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a lifecycle' 2023-01-24T16:52:36.450 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2023-01-24T16:52:36.459 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-01-24T16:52:36.498 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-01-24T16:52:36.498 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T16:52:36.514 INFO:tasks.rgw.client.0.smithi097.stdout:2023-01-24T16:52:36.498+0000 7f12b1fc1700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground (PID: 84871) UID: 0 2023-01-24T16:52:36.514 INFO:tasks.rgw.client.0.smithi097.stdout:2023-01-24T16:52:36.498+0000 7f12c3269740 -1 shutting down 2023-01-24T16:52:42.600 INFO:tasks.rgw.client.0:Stopped 2023-01-24T16:52:42.601 DEBUG:teuthology.orchestra.run.smithi097:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-01-24T16:52:42.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/vault-root-token 2023-01-24T16:52:42.700 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-01-24T16:52:42.709 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-01-24T16:52:42.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T16:52:43.051 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-01-24T16:52:43.051 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":41,"fsid":"ba9bee0e-b74d-4668-a675-133390900940","created":"2023-01-24T16:38:07.892199+0000","modified":"2023-01-24T16:49:15.177968+0000","last_up_change":"2023-01-24T16:38:15.420733+0000","last_in_change":"2023-01-24T16:38:08.842585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T16:38:16.526984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T16:38:17.311816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-01-24T16:38:22.442470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-01-24T16:38:24.629638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-01-24T16:38:26.411163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-01-24T16:38:28.354955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-01-24T16:38:30.342425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-01-24T16:38:32.403944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2023-01-24T16:38:37.808666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2023-01-24T16:47:19.898899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"default.rgw.buckets.data.lukewarm","create_time":"2023-01-24T16:48:21.984043+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"default.rgw.buckets.data.frozen","create_time":"2023-01-24T16:49:12.544402+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"a9a7d912-dc32-4e2c-a692-97890541722c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":79845},{"type":"v1","addr":"172.21.15.97:6809","nonce":79845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":79845},{"type":"v1","addr":"172.21.15.97:6811","nonce":79845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":79845},{"type":"v1","addr":"172.21.15.97:6815","nonce":79845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":79845},{"type":"v1","addr":"172.21.15.97:6813","nonce":79845}]},"public_addr":"172.21.15.97:6809/79845","cluster_addr":"172.21.15.97:6811/79845","heartbeat_back_addr":"172.21.15.97:6815/79845","heartbeat_front_addr":"172.21.15.97:6813/79845","state":["exists","up"]},{"osd":1,"uuid":"3719c1fb-f16b-4fa9-8110-43052f05c318","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":79846},{"type":"v1","addr":"172.21.15.97:6801","nonce":79846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":79846},{"type":"v1","addr":"172.21.15.97:6803","nonce":79846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":79846},{"type":"v1","addr":"172.21.15.97:6807","nonce":79846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":79846},{"type":"v1","addr":"172.21.15.97:6805","nonce":79846}]},"public_addr":"172.21.15.97:6801/79846","cluster_addr":"172.21.15.97:6803/79846","heartbeat_back_addr":"172.21.15.97:6807/79846","heartbeat_front_addr":"172.21.15.97:6805/79846","state":["exists","up"]},{"osd":2,"uuid":"3c859fd8-83e0-478d-8766-276c16357101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":79875},{"type":"v1","addr":"172.21.15.97:6817","nonce":79875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":79875},{"type":"v1","addr":"172.21.15.97:6819","nonce":79875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":79875},{"type":"v1","addr":"172.21.15.97:6823","nonce":79875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":79875},{"type":"v1","addr":"172.21.15.97:6821","nonce":79875}]},"public_addr":"172.21.15.97:6817/79875","cluster_addr":"172.21.15.97:6819/79875","heartbeat_back_addr":"172.21.15.97:6823/79875","heartbeat_front_addr":"172.21.15.97:6821/79875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:38:13.270598+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:38:13.267327+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:38:13.210037+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":1}]},{"pgid":"3.10","mappings":[{"from":0,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":0}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":0}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.33","mappings":[{"from":1,"to":0}]},{"pgid":"4.38","mappings":[{"from":1,"to":2}]},{"pgid":"4.3a","mappings":[{"from":1,"to":0}]},{"pgid":"10.1a","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T16:52:43.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-01-24T16:52:43.377 INFO:teuthology.misc:Shutting down mds daemons... 2023-01-24T16:52:43.378 INFO:teuthology.misc:Shutting down osd daemons... 2023-01-24T16:52:43.378 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-01-24T16:52:43.379 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T16:52:43.423 INFO:tasks.ceph.osd.0:Stopped 2023-01-24T16:52:43.423 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-01-24T16:52:43.424 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T16:52:43.459 INFO:tasks.ceph.osd.1:Stopped 2023-01-24T16:52:43.460 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-01-24T16:52:43.460 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T16:52:43.531 INFO:tasks.ceph.osd.2:Stopped 2023-01-24T16:52:43.532 INFO:teuthology.misc:Shutting down mgr daemons... 2023-01-24T16:52:43.532 DEBUG:tasks.ceph.mgr.0:waiting for process to exit 2023-01-24T16:52:43.533 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T16:52:43.594 INFO:tasks.ceph.mgr.0:Stopped 2023-01-24T16:52:43.594 INFO:teuthology.misc:Shutting down mon daemons... 2023-01-24T16:52:43.595 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-01-24T16:52:43.595 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T16:52:43.645 INFO:tasks.ceph.mon.a:Stopped 2023-01-24T16:52:43.646 INFO:tasks.ceph:Checking cluster log for badness... 2023-01-24T16:52:43.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-01-24T16:52:43.710 INFO:teuthology.orchestra.run.smithi097.stdout:2023-01-24T16:44:36.359081+0000 mon.a (mon.0) 169 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2023-01-24T16:52:43.711 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-01-24T16:52:43.711 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi097.front.sepia.ceph.com 2023-01-24T16:52:43.712 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-01-24T16:52:44.007 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi097.front.sepia.ceph.com 2023-01-24T16:52:44.008 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-01-24T16:52:44.092 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi097.front.sepia.ceph.com 2023-01-24T16:52:44.093 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-01-24T16:52:44.154 INFO:tasks.ceph:Archiving mon data... 2023-01-24T16:52:44.155 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135445/data/mon.a.tgz 2023-01-24T16:52:44.156 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2023-01-24T16:52:44.170 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.Ud6lkXDGx3 2023-01-24T16:52:44.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Ud6lkXDGx3 2023-01-24T16:52:44.308 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.Ud6lkXDGx3 2023-01-24T16:52:44.388 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.Ud6lkXDGx3 is 288KB 2023-01-24T16:52:44.485 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.Ud6lkXDGx3 2023-01-24T16:52:44.500 INFO:tasks.ceph:Cleaning ceph cluster... 2023-01-24T16:52:44.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-01-24T16:52:44.563 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-01-24T16:52:44.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-01-24T16:52:44.628 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-01-24T16:52:44.634 INFO:tasks.ceph:Archiving crash dumps... 2023-01-24T16:52:44.647 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135445/remote/smithi097/crash 2023-01-24T16:52:44.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2023-01-24T16:52:44.714 INFO:tasks.ceph:Compressing logs... 2023-01-24T16:52:44.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-01-24T16:52:59.010 INFO:tasks.ceph:Archiving logs... 2023-01-24T16:52:59.011 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135445/remote/smithi097/log 2023-01-24T16:52:59.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar cz -f - -C /var/log/ceph -- . 2023-01-24T16:53:01.772 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-01-24T16:53:01.781 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2023-01-24T16:53:01.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2023-01-24T16:53:01.869 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-01-24T16:53:01.878 INFO:teuthology.task.clock:Checking final clock skew... 2023-01-24T16:53:01.878 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T16:53:01.892 INFO:teuthology.orchestra.run.smithi097.stderr:bash: ntpq: command not found 2023-01-24T16:53:01.895 INFO:teuthology.orchestra.run.smithi097.stdout:210 Number of sources = 4 2023-01-24T16:53:01.895 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T16:53:01.895 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2023-01-24T16:53:01.896 INFO:teuthology.orchestra.run.smithi097.stdout:^- hv01.front.sepia.ceph.com 3 6 377 38 +5123us[+5141us] +/- 92ms 2023-01-24T16:53:01.896 INFO:teuthology.orchestra.run.smithi097.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -118us[ -100us] +/- 49ms 2023-01-24T16:53:01.896 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +138us[ +156us] +/- 52ms 2023-01-24T16:53:01.897 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T16:53:01.897 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-01-24T16:53:01.988 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-01-24T16:53:01.989 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-01-24T16:53:02.026 DEBUG:teuthology.orchestra.run.smithi097:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-01-24T16:53:02.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T16:53:02.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T16:53:02.137 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-01-24T16:53:02.147 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-01-24T16:53:02.185 INFO:teuthology.task.internal:Duration was 1525.676092 seconds 2023-01-24T16:53:02.220 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-01-24T16:53:02.236 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-01-24T16:53:02.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-01-24T16:53:02.280 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T16:53:02.387 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-01-24T16:53:02.388 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2023-01-24T16:53:02.388 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-01-24T16:53:02.446 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-01-24T16:53:02.447 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-01-24T16:53:02.522 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-01-24T16:53:02.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-01-24T16:53:02.630 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-01-24T16:53:02.643 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-01-24T16:53:02.643 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-01-24T16:53:02.667 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-01-24T16:53:02.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-01-24T16:53:02.761 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2023-01-24T16:53:02.774 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2023-01-24T16:53:02.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T16:53:02.828 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-01-24T16:53:02.869 INFO:teuthology.task.internal:Transferring archived files... 2023-01-24T16:53:02.870 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135445/remote/smithi097 2023-01-24T16:53:02.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2023-01-24T16:53:02.917 INFO:teuthology.task.internal:Removing archive directory... 2023-01-24T16:53:02.918 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2023-01-24T16:53:02.969 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-01-24T16:53:02.980 INFO:teuthology.task.internal:Not uploading archives. 2023-01-24T16:53:02.980 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-01-24T16:53:03.010 INFO:teuthology.task.internal:Tidying up after the test... 2023-01-24T16:53:03.011 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-01-24T16:53:03.024 INFO:teuthology.orchestra.run.smithi097.stdout: 262168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 24 16:53 /home/ubuntu/cephtest 2023-01-24T16:53:03.026 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-01-24T16:53:03.053 INFO:teuthology.nuke:Checking targets against current locks 2023-01-24T16:53:03.084 DEBUG:teuthology.nuke:shortname: smithi097 2023-01-24T16:53:03.085 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-24T16:53:03.111 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135445', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 16:22:25.301309', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDH2P9SGj7EEEyNLxYTnBNzzYgzRuF3LWwwm4A9kPUSZVETbWpQyGzTXkCuC8JGJK6hlTXwbQxL5RzTNGiyXbtWfZ8owkv2lN9OCXtsbG07syuCg0IfNd6/dx67qXikCQYA/mW/2165tx0Lr1tzPApYcOIB6SqduX3cfs5jzos3o+HU6VP9UnmjBd3O6w2aw4RSdaHknpq9CjT9Qm5HuCr1hNlWu9p7OYOXkIYLK9LmNfVQSAA3Rz50THlCBRIlNQg1Boh1RfXChWiBFJ88jEKBm96E2R9oYQXb6FQmBujO5IzktunT/CSgheILfDyOX/Xj/h5j0ZOdJICPq+kr83mzM4dqPzV/b2bJbOWlntqi7JeJz7vZuHWHG3MWiek9RPFxn2FWO4Cv0VpDJ/Ttvq5F8C3TP1dEiMhUB7xpwk1eWSsb8tKAX1cyvHZXC8RPL/TM+qAlTYBR1qrTwNRVZLJf+ygsQ2wSeVw2ZCHbTmoL1q3ML2W+HHhVHifXVSMV5es='} 2023-01-24T16:53:03.164 INFO:teuthology.orchestra.console:Power off smithi097 2023-01-24T16:53:03.165 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-01-24T16:53:03.189 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-01-24T16:53:07.212 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-01-24T16:53:11.324 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-01-24T16:53:15.436 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-01-24T16:53:15.647 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2023-01-24T16:53:15.752 INFO:teuthology.run:Summary data: description: rgw/lifecycle/{cluster overrides supported-random-distro$/{rhel_8} tasks/rgw_s3tests} duration: 1525.6760919094086 failure_reason: 'Command failed (s3 tests against rgw) on smithi097 with status 1: ''S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a lifecycle''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=82e5f46a5c8a43348e2d4e8652604f37 status: fail success: false 2023-01-24T16:53:15.753 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T16:53:15.822 INFO:teuthology.run:FAIL