2023-01-24T17:17:43.097 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-24T17:17:43.098 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135512 --owner scheduled_yuriw@teuthology --name yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi --verbose --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} -- /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135512/orig.config.yaml 2023-01-24T17:17:43.147 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T17:17:43.204 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135512 branch: wip-yuri7-testing-2023-01-23-1532-quincy description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} email: yweinstein@radhat.com first_in_suite: false job_id: '7135512' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri7-testing-2023-01-23-1532-quincy ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: vault rgw crypt sse s3 backend: vault rgw crypt sse s3 vault auth: token rgw crypt sse s3 vault prefix: /v1/transit/ rgw crypt sse s3 vault secret engine: transit rgw crypt vault auth: token rgw crypt vault prefix: /v1/transit/ rgw crypt vault secret engine: transit setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 49f8fb05584886826e8eade75f7105fba754560c wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: default sha1: 49f8fb05584886826e8eade75f7105fba754560c rgw: client.0: use-vault-role: client.0 s3tests: with-sse-s3: true selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri7-testing-2023-01-23-1532-quincy sha1: 49f8fb05584886826e8eade75f7105fba754560c owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 49f8fb05584886826e8eade75f7105fba754560c sleep_before_teardown: 0 suite: rgw suite_branch: wip-yuri7-testing-2023-01-23-1532-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 49f8fb05584886826e8eade75f7105fba754560c targets: smithi150.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGGpG4v9iPw5KlTsygw1ycqMXRVvRKyDYkADHNM6Ff2iNjX/FdHqHMfASQVQ65nHO8FCrp1oZ/aC14oEUjAY0YKCIcFSFjEDmsyVsogDr0oe2JX0KL/F9e829HqFDr1cOv56apRA3bjW+QUxMT1ra09N1HOPKATBA+juGpTW7fj2SL8opgc+8qnY3cAJlA3VQxkPc4EkVeV0ywF9Q7cwFeCJ95Mknj4vbJcNFzZA0DNl4pZkAqPV4imLUm87Wy5c4Oh6xTOuDHHP8kUTJ/2BSkx7LQKR7I0sqd+90ekhvLVOim1rHmTV0x4pvJil3oK0pFsRPiNkTNPzzeiNkHB3arqTmyGTEhwQYyFKxzAA0qeZ6s7hbTp2BkMN+u6dN9MHy1CkD3uEiMnD7/hiKg7m6Dtn890DGXe2SDH+/nHjAgfo4OjI3Haqru+WxHgt2Evhw2GAD+iYGpaXjcXdIw3jeTT8DLotieUadtB46BKPVTP377or8kiu7IyC0PPtXygdc= tasks: - install: null - ceph: null - vault: client.0: engine: transit install_sha256: 7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 install_url: https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip prefix: /v1/transit/keys/ root_token: test_root_token secrets: - path: my-key-1 - path: my-key-2 - rgw: client.0: null - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 force-branch: ceph-quincy vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_16:04:03 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-24T17:17:43.205 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa; will attempt to use it 2023-01-24T17:17:43.205 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa/tasks 2023-01-24T17:17:43.206 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-24T17:17:43.208 INFO:teuthology.task.internal:Checking packages... 2023-01-24T17:17:43.235 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '49f8fb05584886826e8eade75f7105fba754560c' 2023-01-24T17:17:43.235 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-24T17:17:43.236 INFO:teuthology.packaging:ref: None 2023-01-24T17:17:43.236 INFO:teuthology.packaging:tag: None 2023-01-24T17:17:43.236 INFO:teuthology.packaging:branch: wip-yuri7-testing-2023-01-23-1532-quincy 2023-01-24T17:17:43.237 INFO:teuthology.packaging:sha1: 49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T17:17:43.237 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri7-testing-2023-01-23-1532-quincy 2023-01-24T17:17:43.521 INFO:teuthology.task.internal:Found packages for ceph version 17.2.5-727.g49f8fb05 2023-01-24T17:17:43.522 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-24T17:17:43.529 INFO:teuthology.task.internal:no buildpackages task found 2023-01-24T17:17:43.529 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-24T17:17:43.561 INFO:teuthology.task.internal:Saving configuration 2023-01-24T17:17:43.573 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-24T17:17:43.631 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-24T17:17:43.653 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 17:12:42.408618', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN2XERC40E7N4FaZrknnipaL6ypkB87hC8FsCcnN+zuhKK6gL9V9QCGwUBzGPQ4q5+oDbxST7IjH254a51IfZ6k='} 2023-01-24T17:17:43.654 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-24T17:17:43.702 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-01-24T17:17:43.702 INFO:teuthology.run_tasks:Running task console_log... 2023-01-24T17:17:43.759 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9ae39cc5e0>, signals=[15]) 2023-01-24T17:17:43.760 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-24T17:17:43.766 INFO:teuthology.task.internal:Opening connections... 2023-01-24T17:17:43.767 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2023-01-24T17:17:43.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T17:17:43.838 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-24T17:17:43.846 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2023-01-24T17:17:43.863 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2023-01-24T17:17:43.864 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2023-01-24T17:17:43.919 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="Red Hat Enterprise Linux" 2023-01-24T17:17:43.920 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8.4 (Ootpa)" 2023-01-24T17:17:43.920 INFO:teuthology.orchestra.run.smithi150.stdout:ID="rhel" 2023-01-24T17:17:43.920 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="fedora" 2023-01-24T17:17:43.921 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8.4" 2023-01-24T17:17:43.921 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8" 2023-01-24T17:17:43.921 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-01-24T17:17:43.921 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2023-01-24T17:17:43.921 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-01-24T17:17:43.922 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.redhat.com/" 2023-01-24T17:17:43.922 INFO:teuthology.orchestra.run.smithi150.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-01-24T17:17:43.922 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-24T17:17:43.923 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:17:43.923 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-24T17:17:43.923 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-01-24T17:17:43.924 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-01-24T17:17:43.924 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-01-24T17:17:43.925 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-01-24T17:17:43.951 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-24T17:17:43.959 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-24T17:17:43.998 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-24T17:17:43.998 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2023-01-24T17:17:44.015 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-24T17:17:44.059 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-24T17:17:44.060 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2023-01-24T17:17:44.076 INFO:teuthology.orchestra.run.smithi150.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-01-24T17:17:44.077 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-24T17:17:44.128 INFO:teuthology.run_tasks:Running task kernel... 2023-01-24T17:17:44.150 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-24T17:17:44.151 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-24T17:17:44.151 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-24T17:17:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T17:17:44.168 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2023-01-24T17:17:44.224 INFO:teuthology.orchestra.run.smithi150.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-01-24T17:17:44.225 INFO:teuthology.task.kernel:Running kernel on smithi150: 4.18.0-305.25.1.el8_4.x86_64 2023-01-24T17:17:44.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2023-01-24T17:17:45.623 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:17:45.624 INFO:teuthology.orchestra.run.smithi150.stdout:Unable to read consumer identity 2023-01-24T17:17:46.965 INFO:teuthology.orchestra.run.smithi150.stdout:Copr repo for python3-asyncssh owned by ceph 34 kB/s | 3.5 kB 00:00 2023-01-24T17:17:48.684 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 13 MB 00:01 2023-01-24T17:17:51.464 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2023-01-24T17:17:52.138 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-01-24T17:17:52.139 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-01-24T17:17:52.139 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-01-24T17:17:52.267 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:17:52.268 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-01-24T17:17:52.269 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:17:52.372 DEBUG:teuthology.orchestra.run.smithi150:> echo no | sudo yum reinstall kernel || true 2023-01-24T17:17:52.718 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:17:52.718 INFO:teuthology.orchestra.run.smithi150.stdout:Unable to read consumer identity 2023-01-24T17:17:53.014 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Jan 2023 05:17:51 PM UTC. 2023-01-24T17:17:53.150 INFO:teuthology.orchestra.run.smithi150.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-01-24T17:17:53.151 INFO:teuthology.orchestra.run.smithi150.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-01-24T17:17:53.151 INFO:teuthology.orchestra.run.smithi150.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-01-24T17:17:53.180 INFO:teuthology.orchestra.run.smithi150.stderr:Error: No packages marked for reinstall. 2023-01-24T17:17:53.230 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum reinstall -y kernel || true 2023-01-24T17:17:53.574 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:17:53.574 INFO:teuthology.orchestra.run.smithi150.stdout:Unable to read consumer identity 2023-01-24T17:17:53.853 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Jan 2023 05:17:51 PM UTC. 2023-01-24T17:17:54.000 INFO:teuthology.orchestra.run.smithi150.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-01-24T17:17:54.000 INFO:teuthology.orchestra.run.smithi150.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-01-24T17:17:54.000 INFO:teuthology.orchestra.run.smithi150.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-01-24T17:17:54.012 INFO:teuthology.orchestra.run.smithi150.stderr:Error: No packages marked for reinstall. 2023-01-24T17:17:54.067 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel --last | head -n 1 2023-01-24T17:17:54.191 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-4.18.0-305.25.1.el8_4.x86_64 Wed 21 Dec 2022 01:05:24 PM UTC 2023-01-24T17:17:54.193 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.25.1.el8_4.x86_64 2023-01-24T17:17:54.193 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-01-24T17:17:54.193 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-24T17:17:54.194 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-24T17:17:54.226 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2023-01-24T17:17:54.240 DEBUG:teuthology.parallel:result is None 2023-01-24T17:17:54.240 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-24T17:17:54.254 INFO:teuthology.task.internal:Creating test directory... 2023-01-24T17:17:54.254 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-24T17:17:54.294 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-24T17:17:54.304 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-24T17:17:54.327 INFO:teuthology.task.internal:Creating archive directory... 2023-01-24T17:17:54.328 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-24T17:17:54.364 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-24T17:17:54.372 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-24T17:17:54.373 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-24T17:17:54.426 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T17:17:54.435 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T17:17:54.437 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-24T17:17:54.444 INFO:teuthology.task.internal:Configuring sudo... 2023-01-24T17:17:54.445 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-24T17:17:54.501 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-24T17:17:54.510 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-24T17:17:54.510 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-24T17:17:54.555 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T17:17:54.643 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T17:17:54.678 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T17:17:54.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T17:17:54.797 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:17:54.798 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-24T17:17:54.860 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2023-01-24T17:17:54.948 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T17:17:54.965 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-24T17:17:54.974 INFO:teuthology.task.internal:Starting timer... 2023-01-24T17:17:54.974 INFO:teuthology.run_tasks:Running task pcp... 2023-01-24T17:17:55.013 INFO:teuthology.run_tasks:Running task selinux... 2023-01-24T17:17:55.024 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-01-24T17:17:55.025 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2023-01-24T17:17:55.083 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: [ OK ] 2023-01-24T17:17:55.085 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-24T17:17:55.085 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2023-01-24T17:17:55.109 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2023-01-24T17:17:55.109 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi150.front.sepia.ceph.com': 'permissive'} 2023-01-24T17:17:55.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T17:17:55.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T17:17:55.176 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-24T17:17:55.177 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2023-01-24T17:17:55.241 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-24T17:17:55.258 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-01-24T17:17:55.263 INFO:teuthology.repo_utils:Fetching main from origin 2023-01-24T17:17:55.380 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-24T17:17:55.397 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-24T17:17:55.398 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-24T17:24:19.067 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2023-01-24T17:24:19.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-24T17:24:19.070 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T17:24:19.141 DEBUG:teuthology.orchestra.run.smithi150:> true 2023-01-24T17:24:19.208 INFO:teuthology.run_tasks:Running task clock... 2023-01-24T17:24:19.218 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-24T17:24:19.219 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-24T17:24:19.220 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T17:24:19.277 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-24T17:24:19.291 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-24T17:24:19.329 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2023-01-24T17:24:19.339 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2023-01-24T17:24:19.353 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-24T17:24:19.366 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-24T17:24:19.439 INFO:teuthology.orchestra.run.smithi150.stderr:bash: ntpq: command not found 2023-01-24T17:24:19.442 INFO:teuthology.orchestra.run.smithi150.stdout:210 Number of sources = 4 2023-01-24T17:24:19.442 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T17:24:19.443 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2023-01-24T17:24:19.443 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T17:24:19.443 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T17:24:19.443 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T17:24:19.443 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T17:24:19.444 INFO:teuthology.run_tasks:Running task install... 2023-01-24T17:24:19.458 DEBUG:teuthology.task.install:project ceph 2023-01-24T17:24:19.459 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}} 2023-01-24T17:24:19.459 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'} 2023-01-24T17:24:19.459 INFO:teuthology.task.install:Using flavor: default 2023-01-24T17:24:19.465 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-24T17:24:19.466 INFO:teuthology.task.install:extra packages: [] 2023-01-24T17:24:19.466 DEBUG:teuthology.orchestra.run.smithi150:> sudo lsb_release -is 2023-01-24T17:24:19.522 INFO:teuthology.orchestra.run.smithi150.stdout:RedHatEnterprise 2023-01-24T17:24:19.523 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2023-01-24T17:24:19.524 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'tag': None, 'wait_for_package': False} 2023-01-24T17:24:19.524 DEBUG:teuthology.orchestra.run.smithi150:> sudo lsb_release -is 2023-01-24T17:24:19.600 INFO:teuthology.orchestra.run.smithi150.stdout:RedHatEnterprise 2023-01-24T17:24:19.601 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2023-01-24T17:24:19.602 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T17:24:19.749 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/centos/8/flavors/default/ 2023-01-24T17:24:19.749 INFO:teuthology.task.install.rpm:Package version is 17.2.5-727.g49f8fb05 2023-01-24T17:24:19.868 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-24T17:24:19.868 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:24:19.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-24T17:24:19.892 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-01-24T17:24:19.892 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/49f8fb05584886826e8eade75f7105fba754560c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-24T17:24:19.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-24T17:24:20.030 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-24T17:24:20.093 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2023-01-24T17:24:20.094 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2023-01-24T17:24:20.478 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:24:20.782 INFO:teuthology.orchestra.run.smithi150.stdout:44 files removed 2023-01-24T17:24:20.832 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw 2023-01-24T17:24:21.177 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:24:21.852 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 317 kB/s | 78 kB 00:00 2023-01-24T17:24:22.093 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 46 kB/s | 9.2 kB 00:00 2023-01-24T17:24:22.323 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 7.3 kB/s | 1.5 kB 00:00 2023-01-24T17:24:22.425 INFO:teuthology.orchestra.run.smithi150.stdout:Copr repo for python3-asyncssh owned by ceph 47 kB/s | 3.5 kB 00:00 2023-01-24T17:24:26.311 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 3.5 MB/s | 13 MB 00:03 2023-01-24T17:24:29.123 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 1.4 MB/s | 24 kB 00:00 2023-01-24T17:24:29.493 INFO:teuthology.orchestra.run.smithi150.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 130 MB/s | 38 MB 00:00 2023-01-24T17:24:35.002 INFO:teuthology.orchestra.run.smithi150.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 124 MB/s | 33 MB 00:00 2023-01-24T17:24:43.529 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:24:43.532 INFO:teuthology.orchestra.run.smithi150.stdout:=================================================================================================== 2023-01-24T17:24:43.532 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-01-24T17:24:43.532 INFO:teuthology.orchestra.run.smithi150.stdout:=================================================================================================== 2023-01-24T17:24:43.532 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:24:43.533 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 17 M 2023-01-24T17:24:43.533 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2023-01-24T17:24:43.533 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 3.5 M 2023-01-24T17:24:43.533 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 3.6 M 2023-01-24T17:24:43.533 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-01-24T17:24:43.534 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 5.1 M 2023-01-24T17:24:43.534 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 23 M 2023-01-24T17:24:43.534 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 24 k 2023-01-24T17:24:43.534 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-24T17:24:43.534 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 793 k 2023-01-24T17:24:43.534 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-24T17:24:43.535 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-01-24T17:24:43.535 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 506 k 2023-01-24T17:24:43.535 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-01-24T17:24:43.535 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 5.5 M 2023-01-24T17:24:43.535 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-24T17:24:43.536 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-01-24T17:24:43.536 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 45 k 2023-01-24T17:24:43.536 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 108 k 2023-01-24T17:24:43.536 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 190 k 2023-01-24T17:24:43.536 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-01-24T17:24:43.537 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 370 k 2023-01-24T17:24:43.537 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 357 k 2023-01-24T17:24:43.537 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 107 k 2023-01-24T17:24:43.537 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-24T17:24:43.537 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:24:43.538 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:24:43.538 INFO:teuthology.orchestra.run.smithi150.stdout:=================================================================================================== 2023-01-24T17:24:43.538 INFO:teuthology.orchestra.run.smithi150.stdout:Install 21 Packages 2023-01-24T17:24:43.538 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 2 Packages 2023-01-24T17:24:43.538 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:24:43.539 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 63 M 2023-01-24T17:24:43.539 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:24:47.746 INFO:teuthology.orchestra.run.smithi150.stdout:(1/23): ceph-base-17.2.5-727.g49f8fb05.el8.x86_ 1.2 MB/s | 5.1 MB 00:04 2023-01-24T17:24:47.813 INFO:teuthology.orchestra.run.smithi150.stdout:(2/23): ceph-selinux-17.2.5-727.g49f8fb05.el8.x 363 kB/s | 24 kB 00:00 2023-01-24T17:24:48.020 INFO:teuthology.orchestra.run.smithi150.stdout:(3/23): libcephfs2-17.2.5-727.g49f8fb05.el8.x86 3.7 MB/s | 793 kB 00:00 2023-01-24T17:24:48.164 INFO:teuthology.orchestra.run.smithi150.stdout:(4/23): libradosstriper1-17.2.5-727.g49f8fb05.e 3.5 MB/s | 506 kB 00:00 2023-01-24T17:24:49.083 INFO:teuthology.orchestra.run.smithi150.stdout:(5/23): librgw2-17.2.5-727.g49f8fb05.el8.x86_64 6.0 MB/s | 5.5 MB 00:00 2023-01-24T17:24:49.145 INFO:teuthology.orchestra.run.smithi150.stdout:(6/23): python3-ceph-argparse-17.2.5-727.g49f8f 729 kB/s | 45 kB 00:00 2023-01-24T17:24:49.488 INFO:teuthology.orchestra.run.smithi150.stdout:(7/23): ceph-radosgw-17.2.5-727.g49f8fb05.el8.x 2.8 MB/s | 17 MB 00:05 2023-01-24T17:24:49.526 INFO:teuthology.orchestra.run.smithi150.stdout:(8/23): python3-ceph-common-17.2.5-727.g49f8fb0 283 kB/s | 108 kB 00:00 2023-01-24T17:24:49.562 INFO:teuthology.orchestra.run.smithi150.stdout:(9/23): python3-cephfs-17.2.5-727.g49f8fb05.el8 2.5 MB/s | 190 kB 00:00 2023-01-24T17:24:49.611 INFO:teuthology.orchestra.run.smithi150.stdout:(10/23): python3-rados-17.2.5-727.g49f8fb05.el8 4.3 MB/s | 370 kB 00:00 2023-01-24T17:24:49.641 INFO:teuthology.orchestra.run.smithi150.stdout:(11/23): python3-rbd-17.2.5-727.g49f8fb05.el8.x 4.4 MB/s | 357 kB 00:00 2023-01-24T17:24:49.682 INFO:teuthology.orchestra.run.smithi150.stdout:(12/23): python3-rgw-17.2.5-727.g49f8fb05.el8.x 1.5 MB/s | 107 kB 00:00 2023-01-24T17:24:49.875 INFO:teuthology.orchestra.run.smithi150.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 304 kB/s | 59 kB 00:00 2023-01-24T17:24:49.967 INFO:teuthology.orchestra.run.smithi150.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 819 kB/s | 75 kB 00:00 2023-01-24T17:24:49.996 INFO:teuthology.orchestra.run.smithi150.stdout:(15/23): gperftools-libs-2.7-9.el8.x86_64.rpm 862 kB/s | 306 kB 00:00 2023-01-24T17:24:50.026 INFO:teuthology.orchestra.run.smithi150.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.5 MB/s | 47 kB 00:00 2023-01-24T17:24:50.041 INFO:teuthology.orchestra.run.smithi150.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 3.0 MB/s | 44 kB 00:00 2023-01-24T17:24:50.058 INFO:teuthology.orchestra.run.smithi150.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 15 MB/s | 259 kB 00:00 2023-01-24T17:24:50.074 INFO:teuthology.orchestra.run.smithi150.stdout:(19/23): librdkafka-0.11.4-1.el8.x86_64.rpm 21 MB/s | 353 kB 00:00 2023-01-24T17:24:50.270 INFO:teuthology.orchestra.run.smithi150.stdout:(20/23): ceph-common-17.2.5-727.g49f8fb05.el8.x 3.5 MB/s | 23 MB 00:06 2023-01-24T17:24:50.723 INFO:teuthology.orchestra.run.smithi150.stdout:(21/23): librbd1-17.2.5-727.g49f8fb05.el8.x86_6 7.9 MB/s | 3.6 MB 00:00 2023-01-24T17:24:50.728 INFO:teuthology.orchestra.run.smithi150.stdout:(22/23): thrift-0.13.0-2.el8.x86_64.rpm 2.3 MB/s | 1.7 MB 00:00 2023-01-24T17:24:50.772 INFO:teuthology.orchestra.run.smithi150.stdout:(23/23): librados2-17.2.5-727.g49f8fb05.el8.x86 5.0 MB/s | 3.5 MB 00:00 2023-01-24T17:24:50.775 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:24:50.775 INFO:teuthology.orchestra.run.smithi150.stdout:Total 8.7 MB/s | 63 MB 00:07 2023-01-24T17:24:50.778 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:24:50.822 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:24:50.823 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:24:51.558 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:24:51.562 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:24:52.715 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:24:52.913 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-01-24T17:24:53.330 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-01-24T17:24:53.596 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 2/25 2023-01-24T17:24:53.692 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 2/25 2023-01-24T17:24:53.894 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:17.2.5-727.g49f8fb05.el8.x86_64 3/25 2023-01-24T17:24:54.154 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-01-24T17:24:54.388 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 5/25 2023-01-24T17:24:54.768 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 5/25 2023-01-24T17:24:54.954 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 6/25 2023-01-24T17:24:55.068 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 6/25 2023-01-24T17:24:55.237 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/25 2023-01-24T17:24:55.302 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/25 2023-01-24T17:24:55.634 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-01-24T17:24:56.270 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-01-24T17:24:56.437 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:17.2.5-727.g49f8fb05.el8.x86_64 10/25 2023-01-24T17:24:56.504 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:17.2.5-727.g49f8fb05.el8.x86_64 10/25 2023-01-24T17:24:56.656 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:17.2.5-727.g49f8fb05.el8.x 11/25 2023-01-24T17:24:56.840 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:17.2.5-727.g49f8fb05.el8.x86_64 12/25 2023-01-24T17:24:57.104 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:17.2.5-727.g49f8fb05.el8.x86_64 13/25 2023-01-24T17:24:57.299 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:17.2.5-727.g49f8fb05.el8.x86_64 14/25 2023-01-24T17:24:57.478 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:17.2.5-727.g49f8fb05.el8.x86_64 15/25 2023-01-24T17:24:57.551 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:17.2.5-727.g49f8fb05.el8.x86_64 15/25 2023-01-24T17:24:57.713 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-01-24T17:24:57.923 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-01-24T17:24:58.168 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-01-24T17:24:58.379 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:17.2.5-727.g49f8fb05.el8.x86 19/25 2023-01-24T17:25:03.077 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 20/25 2023-01-24T17:25:03.287 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 20/25 2023-01-24T17:25:03.855 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 20/25 2023-01-24T17:25:04.113 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 21/25 2023-01-24T17:25:04.183 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 21/25 2023-01-24T17:25:04.311 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:17.2.5-727.g49f8fb05.el8.x86_64 22/25 2023-01-24T17:25:21.850 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:17.2.5-727.g49f8fb05.el8.x86_64 22/25 2023-01-24T17:25:21.850 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2023-01-24T17:25:21.851 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2023-01-24T17:25:21.851 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /dev 2023-01-24T17:25:21.851 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /run 2023-01-24T17:25:21.851 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2023-01-24T17:25:21.852 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2023-01-24T17:25:21.852 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2023-01-24T17:25:21.852 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2023-01-24T17:25:21.852 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /dev 2023-01-24T17:25:21.853 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:25:23.560 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:17.2.5-727.g49f8fb05.el8.x86_64 23/25 2023-01-24T17:25:23.639 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:17.2.5-727.g49f8fb05.el8.x86_64 23/25 2023-01-24T17:25:23.639 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T17:25:23.869 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T17:25:23.869 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T17:25:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T17:25:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-01-24T17:25:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:All rules containing unresolvable specifiers will be skipped. 2023-01-24T17:25:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:25:24.641 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 1/25 2023-01-24T17:25:24.641 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 2/25 2023-01-24T17:25:24.641 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:17.2.5-727.g49f8fb05.el8.x86_64 3/25 2023-01-24T17:25:24.642 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:17.2.5-727.g49f8fb05.el8.x86_64 4/25 2023-01-24T17:25:24.642 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 5/25 2023-01-24T17:25:24.642 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:17.2.5-727.g49f8fb05.el8.x86_64 6/25 2023-01-24T17:25:24.642 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:17.2.5-727.g49f8fb05.el8.x86_64 7/25 2023-01-24T17:25:24.642 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:17.2.5-727.g49f8fb05.el8.x 8/25 2023-01-24T17:25:24.643 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:17.2.5-727.g49f8fb05.el8.x86 9/25 2023-01-24T17:25:24.643 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:17.2.5-727.g49f8fb05.el8.x86_64 10/25 2023-01-24T17:25:24.644 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:17.2.5-727.g49f8fb05.el8.x86_64 11/25 2023-01-24T17:25:24.644 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:17.2.5-727.g49f8fb05.el8.x86_64 12/25 2023-01-24T17:25:24.644 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:17.2.5-727.g49f8fb05.el8.x86_64 13/25 2023-01-24T17:25:24.644 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-01-24T17:25:24.645 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-01-24T17:25:24.645 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-01-24T17:25:24.645 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-01-24T17:25:24.645 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-01-24T17:25:24.645 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2023-01-24T17:25:24.646 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2023-01-24T17:25:24.646 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/25 2023-01-24T17:25:24.646 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 22/25 2023-01-24T17:25:24.646 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-01-24T17:25:24.646 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 24/25 2023-01-24T17:25:26.057 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T17:25:26.057 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:52 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:25:28.113 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:25:28.217 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:25:28.217 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2023-01-24T17:25:28.217 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.217 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.218 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:25:28.218 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.218 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.218 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.218 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.219 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-24T17:25:28.219 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.219 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-24T17:25:28.219 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-24T17:25:28.219 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.220 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-01-24T17:25:28.220 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.220 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-24T17:25:28.220 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-24T17:25:28.220 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.221 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.221 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.221 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-24T17:25:28.221 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.221 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.222 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:28.222 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-24T17:25:28.222 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:25:28.222 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:25:28.358 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-test 2023-01-24T17:25:28.718 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:25:29.434 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:55 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:25:31.199 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:25:31.200 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================== 2023-01-24T17:25:31.200 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-01-24T17:25:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================== 2023-01-24T17:25:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:25:31.201 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 52 M 2023-01-24T17:25:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-01-24T17:25:31.201 INFO:teuthology.orchestra.run.smithi150.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-01-24T17:25:31.202 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 174 k 2023-01-24T17:25:31.202 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-01-24T17:25:31.202 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-01-24T17:25:31.202 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-01-24T17:25:31.202 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:25:31.203 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:25:31.203 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================== 2023-01-24T17:25:31.203 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2023-01-24T17:25:31.203 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:25:31.204 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 53 M 2023-01-24T17:25:31.204 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 233 M 2023-01-24T17:25:31.204 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:25:31.207 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 16 MB/s | 70 kB 00:00 2023-01-24T17:25:31.244 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 4.2 MB/s | 161 kB 00:00 2023-01-24T17:25:31.259 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 12 MB/s | 187 kB 00:00 2023-01-24T17:25:31.275 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 19 MB/s | 302 kB 00:00 2023-01-24T17:25:31.387 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): libcephsqlite-17.2.5-727.g49f8fb05.el8.x 942 kB/s | 174 kB 00:00 2023-01-24T17:25:34.078 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): ceph-test-17.2.5-727.g49f8fb05.el8.x86_6 18 MB/s | 52 MB 00:02 2023-01-24T17:25:34.081 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:25:34.082 INFO:teuthology.orchestra.run.smithi150.stdout:Total 18 MB/s | 53 MB 00:02 2023-01-24T17:25:34.085 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:25:34.108 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:25:34.109 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:25:34.460 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:25:34.464 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:25:34.993 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:25:35.224 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-01-24T17:25:35.448 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-01-24T17:25:35.528 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-01-24T17:25:35.689 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-01-24T17:25:35.872 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-01-24T17:25:36.015 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:17.2.5-727.g49f8fb05.el8.x86_64 5/6 2023-01-24T17:25:41.000 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:17.2.5-727.g49f8fb05.el8.x86_64 5/6 2023-01-24T17:25:41.263 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:17.2.5-727.g49f8fb05.el8.x86_64 6/6 2023-01-24T17:25:41.529 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-test-2:17.2.5-727.g49f8fb05.el8.x86_64 6/6 2023-01-24T17:25:41.529 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:17.2.5-727.g49f8fb05.el8.x86_64 1/6 2023-01-24T17:25:41.529 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:17.2.5-727.g49f8fb05.el8.x86_64 2/6 2023-01-24T17:25:41.529 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-01-24T17:25:41.530 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-01-24T17:25:41.530 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-01-24T17:25:42.507 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-01-24T17:25:42.507 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:08 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:25:44.343 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:25:44.437 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:25:44.438 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:25:44.438 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: jq-1.5-12.el8.x86_64 2023-01-24T17:25:44.439 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-24T17:25:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.3.3-2.el8.x86_64 2023-01-24T17:25:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-24T17:25:44.440 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:25:44.441 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:25:44.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph 2023-01-24T17:25:44.897 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:25:45.632 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:11 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:25:47.411 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:25:47.414 INFO:teuthology.orchestra.run.smithi150.stdout:====================================================================================================== 2023-01-24T17:25:47.414 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-01-24T17:25:47.414 INFO:teuthology.orchestra.run.smithi150.stdout:====================================================================================================== 2023-01-24T17:25:47.414 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:25:47.415 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 6.4 k 2023-01-24T17:25:47.415 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-01-24T17:25:47.415 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 2.3 M 2023-01-24T17:25:47.415 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 1.6 M 2023-01-24T17:25:47.416 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 233 k 2023-01-24T17:25:47.416 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 3.9 M 2023-01-24T17:25:47.416 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 17 M 2023-01-24T17:25:47.416 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-24T17:25:47.416 INFO:teuthology.orchestra.run.smithi150.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-24T17:25:47.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-24T17:25:47.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-24T17:25:47.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-24T17:25:47.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-24T17:25:47.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-24T17:25:47.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-01-24T17:25:47.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-01-24T17:25:47.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-24T17:25:47.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-24T17:25:47.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-24T17:25:47.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-01-24T17:25:47.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-24T17:25:47.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-24T17:25:47.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-24T17:25:47.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-24T17:25:47.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-24T17:25:47.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-24T17:25:47.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-24T17:25:47.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-01-24T17:25:47.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-24T17:25:47.421 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:25:47.421 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:25:47.421 INFO:teuthology.orchestra.run.smithi150.stdout:====================================================================================================== 2023-01-24T17:25:47.421 INFO:teuthology.orchestra.run.smithi150.stdout:Install 28 Packages 2023-01-24T17:25:47.421 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:25:47.422 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 27 M 2023-01-24T17:25:47.422 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 105 M 2023-01-24T17:25:47.422 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:25:47.718 INFO:teuthology.orchestra.run.smithi150.stdout:(1/28): ceph-17.2.5-727.g49f8fb05.el8.x86_64.rp 21 kB/s | 6.4 kB 00:00 2023-01-24T17:25:48.843 INFO:teuthology.orchestra.run.smithi150.stdout:(2/28): ceph-mgr-17.2.5-727.g49f8fb05.el8.x86_6 1.1 MB/s | 1.6 MB 00:01 2023-01-24T17:25:48.995 INFO:teuthology.orchestra.run.smithi150.stdout:(3/28): ceph-mds-17.2.5-727.g49f8fb05.el8.x86_6 1.5 MB/s | 2.3 MB 00:01 2023-01-24T17:25:49.042 INFO:teuthology.orchestra.run.smithi150.stdout:(4/28): ceph-mgr-modules-core-17.2.5-727.g49f8f 4.9 MB/s | 233 kB 00:00 2023-01-24T17:25:49.153 INFO:teuthology.orchestra.run.smithi150.stdout:(5/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 396 kB/s | 44 kB 00:00 2023-01-24T17:25:49.236 INFO:teuthology.orchestra.run.smithi150.stdout:(6/28): ceph-mon-17.2.5-727.g49f8fb05.el8.x86_6 2.6 MB/s | 3.9 MB 00:01 2023-01-24T17:25:49.237 INFO:teuthology.orchestra.run.smithi150.stdout:(7/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 2.1 MB/s | 185 kB 00:00 2023-01-24T17:25:49.293 INFO:teuthology.orchestra.run.smithi150.stdout:(8/28): python3-cherrypy-18.4.0-1.el8.noarch.rp 6.8 MB/s | 384 kB 00:00 2023-01-24T17:25:49.320 INFO:teuthology.orchestra.run.smithi150.stdout:(9/28): python3-jaraco-6.2-6.el8.noarch.rpm 419 kB/s | 11 kB 00:00 2023-01-24T17:25:49.347 INFO:teuthology.orchestra.run.smithi150.stdout:(10/28): python3-jaraco-functools-2.0-4.el8.noa 659 kB/s | 18 kB 00:00 2023-01-24T17:25:49.374 INFO:teuthology.orchestra.run.smithi150.stdout:(11/28): python3-logutils-0.3.5-11.el8.noarch.r 1.8 MB/s | 49 kB 00:00 2023-01-24T17:25:49.375 INFO:teuthology.orchestra.run.smithi150.stdout:(12/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.2 MB/s | 173 kB 00:00 2023-01-24T17:25:49.404 INFO:teuthology.orchestra.run.smithi150.stdout:(13/28): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2023-01-24T17:25:49.431 INFO:teuthology.orchestra.run.smithi150.stdout:(14/28): python3-portend-2.6-1.el8.noarch.rpm 569 kB/s | 16 kB 00:00 2023-01-24T17:25:49.433 INFO:teuthology.orchestra.run.smithi150.stdout:(15/28): python3-pecan-1.3.2-9.el8.noarch.rpm 4.8 MB/s | 283 kB 00:00 2023-01-24T17:25:49.458 INFO:teuthology.orchestra.run.smithi150.stdout:(16/28): python3-simplegeneric-0.8.1-17.el8.noa 702 kB/s | 19 kB 00:00 2023-01-24T17:25:49.461 INFO:teuthology.orchestra.run.smithi150.stdout:(17/28): python3-singledispatch-3.4.0.3-18.el8. 855 kB/s | 24 kB 00:00 2023-01-24T17:25:49.488 INFO:teuthology.orchestra.run.smithi150.stdout:(18/28): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2023-01-24T17:25:49.489 INFO:teuthology.orchestra.run.smithi150.stdout:(19/28): python3-trustme-0.6.0-4.el8.noarch.rpm 971 kB/s | 27 kB 00:00 2023-01-24T17:25:49.544 INFO:teuthology.orchestra.run.smithi150.stdout:(20/28): python3-waitress-1.2.1-2.el8.1.noarch. 4.2 MB/s | 241 kB 00:00 2023-01-24T17:25:49.551 INFO:teuthology.orchestra.run.smithi150.stdout:(21/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.0 MB/s | 251 kB 00:00 2023-01-24T17:25:49.572 INFO:teuthology.orchestra.run.smithi150.stdout:(22/28): python3-webtest-2.0.33-1.el8.noarch.rp 3.1 MB/s | 85 kB 00:00 2023-01-24T17:25:49.579 INFO:teuthology.orchestra.run.smithi150.stdout:(23/28): python3-zc-lockfile-2.0-2.el8.noarch.r 816 kB/s | 23 kB 00:00 2023-01-24T17:25:49.608 INFO:teuthology.orchestra.run.smithi150.stdout:(24/28): python3-werkzeug-0.12.2-4.el8.noarch.r 13 MB/s | 457 kB 00:00 2023-01-24T17:25:49.609 INFO:teuthology.orchestra.run.smithi150.stdout:(25/28): python3-markupsafe-0.23-19.el8.x86_64. 1.3 MB/s | 39 kB 00:00 2023-01-24T17:25:49.623 INFO:teuthology.orchestra.run.smithi150.stdout:(26/28): python3-pytz-2017.2-9.el8.noarch.rpm 3.7 MB/s | 54 kB 00:00 2023-01-24T17:25:49.624 INFO:teuthology.orchestra.run.smithi150.stdout:(27/28): python3-mako-1.0.6-13.el8.noarch.rpm 9.7 MB/s | 157 kB 00:00 2023-01-24T17:25:50.119 INFO:teuthology.orchestra.run.smithi150.stdout:(28/28): ceph-osd-17.2.5-727.g49f8fb05.el8.x86_ 13 MB/s | 17 MB 00:01 2023-01-24T17:25:50.123 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:25:50.123 INFO:teuthology.orchestra.run.smithi150.stdout:Total 10 MB/s | 27 MB 00:02 2023-01-24T17:25:50.127 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:25:50.165 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:25:50.165 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:25:50.538 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:25:50.542 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:25:51.563 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:25:51.772 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-01-24T17:25:52.036 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-01-24T17:25:52.210 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-01-24T17:25:52.495 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-01-24T17:25:52.870 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-01-24T17:25:53.122 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-01-24T17:25:53.340 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-01-24T17:25:53.538 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 8/28 2023-01-24T17:25:53.687 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-01-24T17:25:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-01-24T17:25:54.037 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-01-24T17:25:54.209 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-01-24T17:25:54.379 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-01-24T17:25:54.600 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-01-24T17:25:54.819 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-01-24T17:25:54.987 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-01-24T17:25:55.261 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-01-24T17:25:55.487 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-01-24T17:25:55.707 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-01-24T17:25:56.035 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-01-24T17:25:56.256 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-01-24T17:25:57.248 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-01-24T17:25:57.626 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:17.2.5-727.g49f8fb05.el8.n 23/28 2023-01-24T17:25:57.797 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 24/28 2023-01-24T17:25:59.280 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 24/28 2023-01-24T17:25:59.431 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:17.2.5-727.g49f8fb05.el8.x86_64 25/28 2023-01-24T17:25:59.824 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:17.2.5-727.g49f8fb05.el8.x86_64 25/28 2023-01-24T17:26:00.057 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:17.2.5-727.g49f8fb05.el8.x86_64 26/28 2023-01-24T17:26:00.274 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:17.2.5-727.g49f8fb05.el8.x86_64 26/28 2023-01-24T17:26:00.454 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:17.2.5-727.g49f8fb05.el8.x86_64 27/28 2023-01-24T17:26:00.472 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:17.2.5-727.g49f8fb05.el8.x86_64 27/28 2023-01-24T17:26:00.570 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:17.2.5-727.g49f8fb05.el8.x86_64 28/28 2023-01-24T17:26:00.981 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-2:17.2.5-727.g49f8fb05.el8.x86_64 28/28 2023-01-24T17:26:00.981 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:17.2.5-727.g49f8fb05.el8.x86_64 1/28 2023-01-24T17:26:00.982 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:17.2.5-727.g49f8fb05.el8.x86_64 2/28 2023-01-24T17:26:00.982 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 3/28 2023-01-24T17:26:00.982 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:17.2.5-727.g49f8fb05.el8.x86_64 4/28 2023-01-24T17:26:00.982 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:17.2.5-727.g49f8fb05.el8.x86_64 5/28 2023-01-24T17:26:00.983 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:17.2.5-727.g49f8fb05.el8.n 6/28 2023-01-24T17:26:00.983 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-01-24T17:26:00.983 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-01-24T17:26:00.983 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-01-24T17:26:00.983 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-01-24T17:26:00.984 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-01-24T17:26:00.984 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-01-24T17:26:00.984 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-01-24T17:26:00.984 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-01-24T17:26:00.984 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-01-24T17:26:00.985 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-01-24T17:26:00.985 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-01-24T17:26:00.985 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-01-24T17:26:00.985 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-01-24T17:26:00.985 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-01-24T17:26:00.986 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 21/28 2023-01-24T17:26:00.986 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-01-24T17:26:00.986 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-01-24T17:26:00.986 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-01-24T17:26:00.986 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-01-24T17:26:00.987 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-01-24T17:26:00.987 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-01-24T17:26:02.400 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-01-24T17:26:02.400 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:28 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:26:04.258 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:26:04.357 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:04.358 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:26:04.358 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:26:04.358 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:26:04.359 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:26:04.359 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:04.359 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:26:04.360 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:26:04.360 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-24T17:26:04.361 INFO:teuthology.orchestra.run.smithi150.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-24T17:26:04.361 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-24T17:26:04.361 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-24T17:26:04.362 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-24T17:26:04.363 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-24T17:26:04.363 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-24T17:26:04.363 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-24T17:26:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-24T17:26:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-24T17:26:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-24T17:26:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-2.6-1.el8.noarch 2023-01-24T17:26:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-24T17:26:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-24T17:26:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-24T17:26:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-24T17:26:04.366 INFO:teuthology.orchestra.run.smithi150.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-24T17:26:04.366 INFO:teuthology.orchestra.run.smithi150.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-24T17:26:04.366 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-24T17:26:04.366 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-24T17:26:04.367 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-24T17:26:04.367 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-24T17:26:04.367 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:04.367 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:26:04.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-base 2023-01-24T17:26:04.864 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:26:05.587 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:31 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:26:06.894 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:26:07.343 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:26:07.344 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-01-24T17:26:07.344 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:26:07.445 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install cephadm 2023-01-24T17:26:07.803 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:26:08.657 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:34 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:26:10.434 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:26:10.435 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:26:10.436 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-01-24T17:26:10.436 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:26:10.436 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:26:10.436 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 78 k 2023-01-24T17:26:10.437 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:10.437 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:26:10.437 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:26:10.437 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-01-24T17:26:10.437 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:10.438 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 78 k 2023-01-24T17:26:10.438 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 355 k 2023-01-24T17:26:10.438 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:26:10.633 INFO:teuthology.orchestra.run.smithi150.stdout:cephadm-17.2.5-727.g49f8fb05.el8.noarch.rpm 402 kB/s | 78 kB 00:00 2023-01-24T17:26:10.633 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:26:10.633 INFO:teuthology.orchestra.run.smithi150.stdout:Total 399 kB/s | 78 kB 00:00 2023-01-24T17:26:10.633 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:26:10.641 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:26:10.641 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:26:10.642 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:26:10.647 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:26:10.699 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:26:13.202 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:26:13.422 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:26:13.626 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:26:15.096 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:26:15.096 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:41 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:26:17.043 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:26:17.138 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:17.138 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:26:17.138 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:17.138 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:17.139 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:26:17.287 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-immutable-object-cache 2023-01-24T17:26:17.658 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:26:18.405 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:44 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:26:20.192 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:26:20.195 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:26:20.196 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-01-24T17:26:20.196 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:26:20.196 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:26:20.196 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 162 k 2023-01-24T17:26:20.197 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:20.197 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:26:20.197 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:26:20.197 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-01-24T17:26:20.197 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:20.198 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 162 k 2023-01-24T17:26:20.198 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 490 k 2023-01-24T17:26:20.198 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:26:20.636 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-immutable-object-cache-17.2.5-727.g49f8fb0 365 kB/s | 162 kB 00:00 2023-01-24T17:26:20.637 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:26:20.637 INFO:teuthology.orchestra.run.smithi150.stdout:Total 364 kB/s | 162 kB 00:00 2023-01-24T17:26:20.638 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:26:20.650 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:26:20.650 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:26:20.704 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:26:20.708 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:26:20.825 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:26:21.003 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:17.2.5-727.g49f8fb05.e 1/1 2023-01-24T17:26:21.426 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.5-727.g49f8fb05.e 1/1 2023-01-24T17:26:22.326 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:17.2.5-727.g49f8fb05.e 1/1 2023-01-24T17:26:22.327 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:48 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:26:24.172 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:26:24.268 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:24.268 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:26:24.268 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:26:24.268 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:24.269 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:26:24.413 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr 2023-01-24T17:26:24.772 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:26:25.503 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:51 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:26:26.825 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:26:27.282 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:26:27.283 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-01-24T17:26:27.283 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:26:27.387 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-dashboard 2023-01-24T17:26:27.745 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:26:28.489 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:54 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:26:30.277 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:26:30.281 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================================== 2023-01-24T17:26:30.281 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-01-24T17:26:30.281 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================================== 2023-01-24T17:26:30.282 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:26:30.282 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 1.6 M 2023-01-24T17:26:30.282 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-01-24T17:26:30.282 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 23 k 2023-01-24T17:26:30.282 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 14 k 2023-01-24T17:26:30.283 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-01-24T17:26:30.283 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-24T17:26:30.283 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-24T17:26:30.283 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:30.283 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:26:30.284 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================================== 2023-01-24T17:26:30.284 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2023-01-24T17:26:30.284 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:30.284 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.0 M 2023-01-24T17:26:30.285 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 7.8 M 2023-01-24T17:26:30.285 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:26:30.475 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): ceph-grafana-dashboards-17.2.5-727.g49f8 119 kB/s | 23 kB 00:00 2023-01-24T17:26:30.481 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): ceph-prometheus-alerts-17.2.5-727.g49f8f 72 kB/s | 14 kB 00:00 2023-01-24T17:26:30.588 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 297 kB/s | 34 kB 00:00 2023-01-24T17:26:30.616 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.5 MB/s | 43 kB 00:00 2023-01-24T17:26:30.946 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): ceph-mgr-dashboard-17.2.5-727.g49f8fb05. 2.5 MB/s | 1.6 MB 00:00 2023-01-24T17:26:31.025 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 361 kB/s | 196 kB 00:00 2023-01-24T17:26:31.028 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:26:31.029 INFO:teuthology.orchestra.run.smithi150.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2023-01-24T17:26:31.031 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:26:31.040 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:26:31.040 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:26:31.347 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:26:31.351 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:26:31.756 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:26:31.970 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-01-24T17:26:32.201 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-01-24T17:26:32.352 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-01-24T17:26:32.490 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:17.2.5-727.g49f8fb05.el8.no 4/6 2023-01-24T17:26:32.828 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:17.2.5-727.g49f8fb05.el8.n 5/6 2023-01-24T17:26:33.092 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:17.2.5-727.g49f8fb05.el8.noarch 6/6 2023-01-24T17:26:33.357 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.5-727.g49f8fb05.el8.noarch 6/6 2023-01-24T17:26:33.357 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:17.2.5-727.g49f8fb05.el8.n 1/6 2023-01-24T17:26:33.357 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:17.2.5-727.g49f8fb05.el8.noarch 2/6 2023-01-24T17:26:33.358 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:17.2.5-727.g49f8fb05.el8.no 3/6 2023-01-24T17:26:33.358 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-01-24T17:26:33.358 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-01-24T17:26:34.249 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-01-24T17:26:34.249 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:00 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:26:36.157 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:26:36.255 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:36.255 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:26:36.256 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:36.256 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:36.256 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:36.256 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-24T17:26:36.257 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-24T17:26:36.257 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-24T17:26:36.257 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:36.257 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:26:36.373 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-diskprediction-local 2023-01-24T17:26:36.732 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:26:37.448 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:03 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:26:39.234 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:26:39.236 INFO:teuthology.orchestra.run.smithi150.stdout:====================================================================================================================== 2023-01-24T17:26:39.236 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-01-24T17:26:39.236 INFO:teuthology.orchestra.run.smithi150.stdout:====================================================================================================================== 2023-01-24T17:26:39.236 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:26:39.236 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 7.4 M 2023-01-24T17:26:39.237 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-01-24T17:26:39.237 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-01-24T17:26:39.237 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-01-24T17:26:39.238 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-01-24T17:26:39.238 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-01-24T17:26:39.238 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-01-24T17:26:39.238 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-01-24T17:26:39.238 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-01-24T17:26:39.239 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:39.239 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:26:39.239 INFO:teuthology.orchestra.run.smithi150.stdout:====================================================================================================================== 2023-01-24T17:26:39.239 INFO:teuthology.orchestra.run.smithi150.stdout:Install 8 Packages 2023-01-24T17:26:39.239 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:39.240 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 36 M 2023-01-24T17:26:39.240 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 199 M 2023-01-24T17:26:39.241 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:26:39.272 INFO:teuthology.orchestra.run.smithi150.stdout:(1/8): libquadmath-8.4.1-1.el8.x86_64.rpm 4.9 MB/s | 169 kB 00:00 2023-01-24T17:26:39.280 INFO:teuthology.orchestra.run.smithi150.stdout:(2/8): libgfortran-8.4.1-1.el8.x86_64.rpm 15 MB/s | 639 kB 00:00 2023-01-24T17:26:39.298 INFO:teuthology.orchestra.run.smithi150.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 13 MB/s | 225 kB 00:00 2023-01-24T17:26:39.363 INFO:teuthology.orchestra.run.smithi150.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 59 MB/s | 3.7 MB 00:00 2023-01-24T17:26:39.438 INFO:teuthology.orchestra.run.smithi150.stdout:(5/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 63 MB/s | 4.7 MB 00:00 2023-01-24T17:26:39.564 INFO:teuthology.orchestra.run.smithi150.stdout:(6/8): python3-scipy-1.0.0-20.module+el8.1.0+33 50 MB/s | 15 MB 00:00 2023-01-24T17:26:39.605 INFO:teuthology.orchestra.run.smithi150.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 28 MB/s | 4.6 MB 00:00 2023-01-24T17:26:40.017 INFO:teuthology.orchestra.run.smithi150.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.5-727 9.5 MB/s | 7.4 MB 00:00 2023-01-24T17:26:40.019 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:26:40.019 INFO:teuthology.orchestra.run.smithi150.stdout:Total 46 MB/s | 36 MB 00:00 2023-01-24T17:26:40.028 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:26:40.047 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:26:40.048 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:26:40.468 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:26:40.472 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:26:41.091 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:26:41.307 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-01-24T17:26:41.442 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-01-24T17:26:41.590 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-01-24T17:26:42.137 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-01-24T17:26:42.298 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-01-24T17:26:42.838 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-01-24T17:26:43.056 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-01-24T17:26:43.789 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-01-24T17:26:44.109 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-01-24T17:26:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-01-24T17:26:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-01-24T17:26:47.668 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-01-24T17:26:47.947 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.5-727.g49f8fb05 8/8 2023-01-24T17:26:48.245 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.5-727.g49f8fb05 8/8 2023-01-24T17:26:48.245 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.5-727.g49f8fb05 1/8 2023-01-24T17:26:48.246 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 2/8 2023-01-24T17:26:48.246 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 3/8 2023-01-24T17:26:48.246 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 4/8 2023-01-24T17:26:48.246 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2023-01-24T17:26:48.246 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2023-01-24T17:26:48.247 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 7/8 2023-01-24T17:26:49.162 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 8/8 2023-01-24T17:26:49.162 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:15 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:26:51.012 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:26:51.110 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:51.110 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:26:51.110 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:51.111 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-01-24T17:26:51.111 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-01-24T17:26:51.111 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.12-1.el8.x86_64 2023-01-24T17:26:51.111 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-01-24T17:26:51.111 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-01-24T17:26:51.112 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-01-24T17:26:51.112 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-01-24T17:26:51.112 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:51.112 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:26:51.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-rook 2023-01-24T17:26:51.617 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:26:52.399 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:18 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:26:54.156 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:26:54.157 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================================= 2023-01-24T17:26:54.158 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-01-24T17:26:54.158 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================================= 2023-01-24T17:26:54.158 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:26:54.158 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 47 k 2023-01-24T17:26:54.158 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-01-24T17:26:54.159 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-24T17:26:54.159 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-24T17:26:54.159 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-24T17:26:54.159 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-01-24T17:26:54.159 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-01-24T17:26:54.160 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-24T17:26:54.160 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-01-24T17:26:54.160 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-01-24T17:26:54.160 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-01-24T17:26:54.160 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-01-24T17:26:54.161 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-24T17:26:54.161 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-24T17:26:54.161 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:54.161 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:26:54.161 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================================= 2023-01-24T17:26:54.162 INFO:teuthology.orchestra.run.smithi150.stdout:Install 13 Packages 2023-01-24T17:26:54.162 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:26:54.162 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 1.9 M 2023-01-24T17:26:54.163 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 25 M 2023-01-24T17:26:54.163 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:26:54.242 INFO:teuthology.orchestra.run.smithi150.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 200 kB/s | 16 kB 00:00 2023-01-24T17:26:54.248 INFO:teuthology.orchestra.run.smithi150.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 378 kB/s | 33 kB 00:00 2023-01-24T17:26:54.300 INFO:teuthology.orchestra.run.smithi150.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.4 MB/s | 81 kB 00:00 2023-01-24T17:26:54.328 INFO:teuthology.orchestra.run.smithi150.stdout:(4/13): python3-rsa-4.8-1.el8.noarch.rpm 2.1 MB/s | 61 kB 00:00 2023-01-24T17:26:54.357 INFO:teuthology.orchestra.run.smithi150.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 2.1 MB/s | 61 kB 00:00 2023-01-24T17:26:54.385 INFO:teuthology.orchestra.run.smithi150.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 5.3 MB/s | 155 kB 00:00 2023-01-24T17:26:54.400 INFO:teuthology.orchestra.run.smithi150.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 2.9 MB/s | 43 kB 00:00 2023-01-24T17:26:54.412 INFO:teuthology.orchestra.run.smithi150.stdout:(8/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 2.2 MB/s | 27 kB 00:00 2023-01-24T17:26:54.425 INFO:teuthology.orchestra.run.smithi150.stdout:(9/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 9.4 MB/s | 126 kB 00:00 2023-01-24T17:26:54.438 INFO:teuthology.orchestra.run.smithi150.stdout:(10/13): python3-pyasn1-modules-0.3.7-6.el8.noa 8.3 MB/s | 110 kB 00:00 2023-01-24T17:26:54.451 INFO:teuthology.orchestra.run.smithi150.stdout:(11/13): python3-jsonpointer-1.10-11.el8.noarch 1.5 MB/s | 20 kB 00:00 2023-01-24T17:26:54.510 INFO:teuthology.orchestra.run.smithi150.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 4.4 MB/s | 1.2 MB 00:00 2023-01-24T17:26:54.574 INFO:teuthology.orchestra.run.smithi150.stdout:(13/13): ceph-mgr-rook-17.2.5-727.g49f8fb05.el8 114 kB/s | 47 kB 00:00 2023-01-24T17:26:54.575 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:26:54.575 INFO:teuthology.orchestra.run.smithi150.stdout:Total 4.6 MB/s | 1.9 MB 00:00 2023-01-24T17:26:54.581 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:26:54.594 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:26:54.594 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:26:54.684 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:26:54.688 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:26:55.207 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:26:55.493 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-01-24T17:26:55.729 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.8-1.el8.noarch 2/13 2023-01-24T17:26:55.954 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-01-24T17:26:56.147 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-01-24T17:26:56.394 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-01-24T17:26:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-01-24T17:26:56.954 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-01-24T17:26:57.205 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-01-24T17:26:57.407 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-01-24T17:26:57.672 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-01-24T17:26:58.870 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-01-24T17:26:59.182 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-01-24T17:26:59.304 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:17.2.5-727.g49f8fb05.el8.noarch 13/13 2023-01-24T17:26:59.573 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.5-727.g49f8fb05.el8.noarch 13/13 2023-01-24T17:26:59.573 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:17.2.5-727.g49f8fb05.el8.noarch 1/13 2023-01-24T17:26:59.573 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-01-24T17:26:59.574 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-01-24T17:26:59.574 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-01-24T17:26:59.574 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-01-24T17:26:59.574 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 6/13 2023-01-24T17:26:59.574 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-01-24T17:26:59.575 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-01-24T17:26:59.575 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-01-24T17:26:59.575 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2023-01-24T17:26:59.575 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2023-01-24T17:26:59.575 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/13 2023-01-24T17:27:00.560 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-01-24T17:27:00.560 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:26 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:02.563 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:27:02.665 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:02.666 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:27:02.666 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:27:02.666 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-24T17:27:02.666 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-24T17:27:02.667 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-24T17:27:02.667 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-24T17:27:02.667 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-24T17:27:02.667 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-24T17:27:02.667 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-24T17:27:02.668 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-24T17:27:02.668 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-24T17:27:02.668 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-24T17:27:02.668 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-24T17:27:02.668 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-24T17:27:02.669 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:02.669 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:27:02.826 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-cephadm 2023-01-24T17:27:03.191 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:27:03.931 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:29 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:05.747 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:27:05.749 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================== 2023-01-24T17:27:05.749 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-01-24T17:27:05.750 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================== 2023-01-24T17:27:05.750 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:27:05.751 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 115 k 2023-01-24T17:27:05.751 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-01-24T17:27:05.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-24T17:27:05.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-01-24T17:27:05.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-01-24T17:27:05.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-24T17:27:05.753 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:05.753 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:27:05.753 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================== 2023-01-24T17:27:05.754 INFO:teuthology.orchestra.run.smithi150.stdout:Install 5 Packages 2023-01-24T17:27:05.754 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:05.755 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 5.9 M 2023-01-24T17:27:05.755 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 26 M 2023-01-24T17:27:05.755 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:27:05.873 INFO:teuthology.orchestra.run.smithi150.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 493 kB/s | 60 kB 00:00 2023-01-24T17:27:05.948 INFO:teuthology.orchestra.run.smithi150.stdout:(2/5): python3-babel-2.5.1-5.el8.noarch.rpm 64 MB/s | 4.8 MB 00:00 2023-01-24T17:27:05.966 INFO:teuthology.orchestra.run.smithi150.stdout:(3/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 29 MB/s | 538 kB 00:00 2023-01-24T17:27:06.078 INFO:teuthology.orchestra.run.smithi150.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.4 MB/s | 476 kB 00:00 2023-01-24T17:27:06.309 INFO:teuthology.orchestra.run.smithi150.stdout:(5/5): ceph-mgr-cephadm-17.2.5-727.g49f8fb05.el 206 kB/s | 115 kB 00:00 2023-01-24T17:27:06.310 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:27:06.311 INFO:teuthology.orchestra.run.smithi150.stdout:Total 11 MB/s | 5.9 MB 00:00 2023-01-24T17:27:06.314 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:27:06.324 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:27:06.324 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:27:06.418 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:27:06.422 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:27:07.435 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:27:07.793 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-01-24T17:27:08.083 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-01-24T17:27:08.420 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-01-24T17:27:08.699 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-01-24T17:27:08.894 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 5/5 2023-01-24T17:27:09.162 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 5/5 2023-01-24T17:27:09.162 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/5 2023-01-24T17:27:09.162 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-01-24T17:27:09.162 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-01-24T17:27:09.163 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-01-24T17:27:10.069 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-01-24T17:27:10.070 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:36 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:12.034 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:27:12.131 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:12.131 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:27:12.131 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:27:12.131 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-24T17:27:12.132 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.5.1-5.el8.noarch 2023-01-24T17:27:12.132 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-01-24T17:27:12.132 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-24T17:27:12.132 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:12.132 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:27:12.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-fuse 2023-01-24T17:27:12.626 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:27:13.368 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:39 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:15.155 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:27:15.156 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:27:15.156 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-01-24T17:27:15.156 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:27:15.157 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:27:15.157 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse 2023-01-24T17:27:15.157 INFO:teuthology.orchestra.run.smithi150.stdout: x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 872 k 2023-01-24T17:27:15.157 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-01-24T17:27:15.157 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-01-24T17:27:15.158 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:15.158 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:27:15.158 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:27:15.158 INFO:teuthology.orchestra.run.smithi150.stdout:Install 2 Packages 2023-01-24T17:27:15.158 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:15.159 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 954 k 2023-01-24T17:27:15.159 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 3.0 M 2023-01-24T17:27:15.160 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:27:15.186 INFO:teuthology.orchestra.run.smithi150.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.8 MB/s | 83 kB 00:00 2023-01-24T17:27:15.750 INFO:teuthology.orchestra.run.smithi150.stdout:(2/2): ceph-fuse-17.2.5-727.g49f8fb05.el8.x86_6 1.4 MB/s | 872 kB 00:00 2023-01-24T17:27:15.751 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:27:15.751 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.6 MB/s | 954 kB 00:00 2023-01-24T17:27:15.753 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:27:15.775 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:27:15.775 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:27:15.859 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:27:15.864 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:27:16.039 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:27:16.278 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-01-24T17:27:16.477 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 2/2 2023-01-24T17:27:16.845 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 2/2 2023-01-24T17:27:16.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 1/2 2023-01-24T17:27:17.753 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-01-24T17:27:17.754 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:43 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:19.739 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:27:19.836 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:19.836 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:27:19.837 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-01-24T17:27:19.837 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:19.837 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:27:20.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-volume 2023-01-24T17:27:21.064 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:27:21.858 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:47 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:23.669 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:27:23.670 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:27:23.670 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-01-24T17:27:23.670 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:27:23.670 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:27:23.670 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 263 k 2023-01-24T17:27:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:23.671 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:27:23.671 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:27:23.671 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-01-24T17:27:23.671 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:23.672 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 263 k 2023-01-24T17:27:23.672 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 1.3 M 2023-01-24T17:27:23.672 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:27:24.230 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-volume-17.2.5-727.g49f8fb05.el8.noarch.rpm 469 kB/s | 263 kB 00:00 2023-01-24T17:27:24.231 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:27:24.231 INFO:teuthology.orchestra.run.smithi150.stdout:Total 468 kB/s | 263 kB 00:00 2023-01-24T17:27:24.232 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:27:24.242 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:27:24.242 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:27:24.263 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:27:24.267 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:27:24.447 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:27:24.693 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:27:25.116 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:27:26.052 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:27:26.052 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:52 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:27.931 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:27:28.026 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:28.027 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:27:28.027 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:27:28.027 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:28.027 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:27:28.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados-devel 2023-01-24T17:27:28.521 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:27:29.253 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:55 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:31.012 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:27:31.013 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:27:31.014 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-01-24T17:27:31.014 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:27:31.014 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:27:31.014 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 130 k 2023-01-24T17:27:31.014 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:31.015 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:27:31.015 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:27:31.015 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-01-24T17:27:31.015 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:31.015 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 130 k 2023-01-24T17:27:31.016 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 464 k 2023-01-24T17:27:31.016 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:27:31.178 INFO:teuthology.orchestra.run.smithi150.stdout:librados-devel-17.2.5-727.g49f8fb05.el8.x86_64. 790 kB/s | 130 kB 00:00 2023-01-24T17:27:31.179 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:27:31.180 INFO:teuthology.orchestra.run.smithi150.stdout:Total 783 kB/s | 130 kB 00:00 2023-01-24T17:27:31.180 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:27:31.186 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:27:31.186 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:27:31.242 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:27:31.246 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:27:31.375 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:27:31.599 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:31.874 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:32.909 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:32.909 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:58 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:34.729 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:27:34.826 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:34.826 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:27:34.826 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:27:34.827 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:34.827 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:27:34.968 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs2 2023-01-24T17:27:35.337 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:27:36.087 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:02 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:37.399 INFO:teuthology.orchestra.run.smithi150.stdout:Package libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:37.856 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:27:37.857 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-01-24T17:27:37.857 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:27:37.970 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs-devel 2023-01-24T17:27:38.334 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:27:39.107 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:05 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:40.860 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:27:40.861 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:27:40.861 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-01-24T17:27:40.861 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:27:40.861 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:27:40.861 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 25 k 2023-01-24T17:27:40.862 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:40.862 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:27:40.862 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:27:40.862 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-01-24T17:27:40.862 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:40.863 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 25 k 2023-01-24T17:27:40.863 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 109 k 2023-01-24T17:27:40.863 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:27:40.986 INFO:teuthology.orchestra.run.smithi150.stdout:libcephfs-devel-17.2.5-727.g49f8fb05.el8.x86_64 200 kB/s | 25 kB 00:00 2023-01-24T17:27:40.986 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:27:40.987 INFO:teuthology.orchestra.run.smithi150.stdout:Total 197 kB/s | 25 kB 00:00 2023-01-24T17:27:40.987 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:27:40.990 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:27:40.991 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:27:40.992 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:27:40.997 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:27:41.050 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:27:41.198 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:41.523 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:42.450 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:42.450 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:08 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:44.284 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:27:44.381 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:44.381 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:27:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:27:44.382 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:27:44.382 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:27:44.524 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados2 2023-01-24T17:27:44.892 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:27:45.624 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:11 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:46.940 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:47.388 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:27:47.389 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-01-24T17:27:47.389 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:27:47.507 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librbd1 2023-01-24T17:27:47.872 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:27:48.666 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:14 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:50.023 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:50.481 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:27:50.482 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-01-24T17:27:50.482 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:27:50.598 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rados 2023-01-24T17:27:50.965 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:27:51.745 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:17 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:53.139 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rados-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:53.592 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:27:53.593 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-01-24T17:27:53.593 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:27:53.698 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rgw 2023-01-24T17:27:54.062 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:27:54.808 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:20 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:56.172 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rgw-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:56.633 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:27:56.635 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-01-24T17:27:56.635 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:27:56.747 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-cephfs 2023-01-24T17:27:57.112 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:27:57.843 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:23 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:27:59.155 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-cephfs-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:59.605 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:27:59.607 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-01-24T17:27:59.607 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:27:59.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rbd 2023-01-24T17:28:00.075 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:28:00.856 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:26 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:28:02.205 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rbd-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:28:02.659 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:28:02.660 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-01-24T17:28:02.660 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:28:02.773 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-fuse 2023-01-24T17:28:03.140 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:28:03.872 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:29 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:28:05.702 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:28:05.703 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:28:05.703 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-01-24T17:28:05.704 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:28:05.704 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:28:05.704 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 90 k 2023-01-24T17:28:05.704 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:05.704 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:28:05.705 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:28:05.705 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-01-24T17:28:05.705 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:05.705 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 90 k 2023-01-24T17:28:05.705 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 264 k 2023-01-24T17:28:05.706 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:28:06.007 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-fuse-17.2.5-727.g49f8fb05.el8.x86_64.rpm 297 kB/s | 90 kB 00:00 2023-01-24T17:28:06.008 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:28:06.009 INFO:teuthology.orchestra.run.smithi150.stdout:Total 296 kB/s | 90 kB 00:00 2023-01-24T17:28:06.009 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:28:06.020 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:28:06.020 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:28:06.076 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:28:06.080 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:28:06.197 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:28:06.396 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:28:06.691 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:28:07.606 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:28:07.607 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:33 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:28:09.501 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:28:09.596 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:09.597 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:28:09.597 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:28:09.597 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:09.598 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:28:09.722 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-mirror 2023-01-24T17:28:10.093 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:28:10.856 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:36 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:28:12.670 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:28:12.672 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:28:12.672 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-01-24T17:28:12.672 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:28:12.672 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:28:12.672 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 3.5 M 2023-01-24T17:28:12.673 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:12.673 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:28:12.673 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:28:12.673 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-01-24T17:28:12.673 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:12.673 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 3.5 M 2023-01-24T17:28:12.674 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 16 M 2023-01-24T17:28:12.674 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:28:13.956 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-mirror-17.2.5-727.g49f8fb05.el8.x86_64.rpm 2.7 MB/s | 3.5 MB 00:01 2023-01-24T17:28:13.957 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:28:13.958 INFO:teuthology.orchestra.run.smithi150.stdout:Total 2.7 MB/s | 3.5 MB 00:01 2023-01-24T17:28:13.958 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:28:13.972 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:28:13.972 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:28:14.040 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:28:14.044 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:28:14.488 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:28:14.679 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:28:15.098 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:28:16.022 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:28:16.022 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:42 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:28:17.925 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:28:18.020 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:18.020 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:28:18.020 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:28:18.020 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:18.021 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:28:18.154 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-nbd 2023-01-24T17:28:18.526 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:28:19.253 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:45 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:28:21.121 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:28:21.121 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:28:21.122 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-01-24T17:28:21.122 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:28:21.122 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:28:21.122 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 172 k 2023-01-24T17:28:21.123 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:21.123 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:28:21.123 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:28:21.123 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-01-24T17:28:21.123 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:21.124 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 172 k 2023-01-24T17:28:21.124 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 548 k 2023-01-24T17:28:21.124 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:28:21.345 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-nbd-17.2.5-727.g49f8fb05.el8.x86_64.rpm 774 kB/s | 172 kB 00:00 2023-01-24T17:28:21.346 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:28:21.346 INFO:teuthology.orchestra.run.smithi150.stdout:Total 769 kB/s | 172 kB 00:00 2023-01-24T17:28:21.347 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:28:21.359 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:28:21.359 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:28:21.415 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:28:21.419 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:28:21.537 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:28:21.709 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:28:22.004 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-nbd-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:28:22.922 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:28:22.923 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:48 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:28:24.859 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:28:24.954 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:24.955 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:28:24.955 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:28:24.955 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:24.955 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:28:25.105 DEBUG:teuthology.parallel:result is None 2023-01-24T17:28:25.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo lsb_release -is 2023-01-24T17:28:25.143 INFO:teuthology.orchestra.run.smithi150.stdout:RedHatEnterprise 2023-01-24T17:28:25.144 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2023-01-24T17:28:25.145 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T17:28:25.473 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-24T17:28:25.492 INFO:teuthology.orchestra.run.smithi150.stdout:17.2.5-727.g49f8fb05.el8 2023-01-24T17:28:25.493 INFO:teuthology.packaging:The installed version of ceph is 17.2.5-727.g49f8fb05.el8 2023-01-24T17:28:25.494 INFO:teuthology.task.install:The correct ceph version 17.2.5-727.g49f8fb05 is installed. 2023-01-24T17:28:25.495 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-24T17:28:25.495 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:25.496 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-24T17:28:25.556 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-24T17:28:25.556 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:25.557 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2023-01-24T17:28:25.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-24T17:28:25.685 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-24T17:28:25.686 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:25.686 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-24T17:28:25.749 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-24T17:28:25.812 INFO:teuthology.run_tasks:Running task ceph... 2023-01-24T17:28:25.899 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-01-24T17:28:25.901 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /var/log/ceph 2023-01-24T17:28:25.925 INFO:tasks.ceph:Disabling ceph logrotate... 2023-01-24T17:28:25.925 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-24T17:28:25.989 INFO:tasks.ceph:Creating extra log directories... 2023-01-24T17:28:25.989 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-24T17:28:26.061 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-01-24T17:28:26.061 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt sse s3 backend': 'vault', 'rgw crypt sse s3 vault auth': 'token', 'rgw crypt sse s3 vault prefix': '/v1/transit/', 'rgw crypt sse s3 vault secret engine': 'transit', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-01-24T17:28:26.061 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135512', 'branch': 'wip-yuri7-testing-2023-01-23-1532-quincy', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}}', 'email': 'yweinstein@radhat.com', 'first_in_suite': False, 'job_id': '7135512', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri7-testing-2023-01-23-1532-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt sse s3 backend': 'vault', 'rgw crypt sse s3 vault auth': 'token', 'rgw crypt sse s3 vault prefix': '/v1/transit/', 'rgw crypt sse s3 vault secret engine': 'transit', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}}, 'rgw': {'client.0': {'use-vault-role': 'client.0'}}, 's3tests': {'with-sse-s3': True}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri7-testing-2023-01-23-1532-quincy', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuri7-testing-2023-01-23-1532-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'targets': {'smithi150.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGGpG4v9iPw5KlTsygw1ycqMXRVvRKyDYkADHNM6Ff2iNjX/FdHqHMfASQVQ65nHO8FCrp1oZ/aC14oEUjAY0YKCIcFSFjEDmsyVsogDr0oe2JX0KL/F9e829HqFDr1cOv56apRA3bjW+QUxMT1ra09N1HOPKATBA+juGpTW7fj2SL8opgc+8qnY3cAJlA3VQxkPc4EkVeV0ywF9Q7cwFeCJ95Mknj4vbJcNFzZA0DNl4pZkAqPV4imLUm87Wy5c4Oh6xTOuDHHP8kUTJ/2BSkx7LQKR7I0sqd+90ekhvLVOim1rHmTV0x4pvJil3oK0pFsRPiNkTNPzzeiNkHB3arqTmyGTEhwQYyFKxzAA0qeZ6s7hbTp2BkMN+u6dN9MHy1CkD3uEiMnD7/hiKg7m6Dtn890DGXe2SDH+/nHjAgfo4OjI3Haqru+WxHgt2Evhw2GAD+iYGpaXjcXdIw3jeTT8DLotieUadtB46BKPVTP377or8kiu7IyC0PPtXygdc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'vault': {'client.0': {'engine': 'transit', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/transit/keys/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1'}, {'path': 'my-key-2'}]}}}, {'rgw': {'client.0': None}}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8e1bba12e584b7ae912e92e94afe9c075ab51fa3', 'timestamp': '2023-01-24_16:04:03', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.17735'} 2023-01-24T17:28:26.062 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-24T17:28:26.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/run/ceph 2023-01-24T17:28:26.171 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:26.172 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2023-01-24T17:28:26.226 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-24T17:28:26.226 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2023-01-24T17:28:26.281 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-24T17:28:26.281 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T17:28:26.281 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 202514 Links: 1 2023-01-24T17:28:26.282 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T17:28:26.282 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T17:28:26.282 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-01-24 17:28:14.868222716 +0000 2023-01-24T17:28:26.282 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-01-24 17:23:50.182445703 +0000 2023-01-24T17:28:26.282 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-01-24 17:23:50.182445703 +0000 2023-01-24T17:28:26.283 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-01-24T17:28:26.283 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-24T17:28:26.343 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-01-24T17:28:26.343 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-01-24T17:28:26.344 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000230832 s, 2.2 MB/s 2023-01-24T17:28:26.345 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-24T17:28:26.401 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2023-01-24T17:28:26.455 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-24T17:28:26.455 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T17:28:26.455 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 202634 Links: 1 2023-01-24T17:28:26.456 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T17:28:26.456 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T17:28:26.456 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-01-24 17:28:14.868222716 +0000 2023-01-24T17:28:26.456 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-01-24 17:23:50.686435862 +0000 2023-01-24T17:28:26.457 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-01-24 17:23:50.686435862 +0000 2023-01-24T17:28:26.457 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-01-24T17:28:26.457 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-24T17:28:26.517 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-01-24T17:28:26.517 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-01-24T17:28:26.517 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000198116 s, 2.6 MB/s 2023-01-24T17:28:26.518 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-24T17:28:26.573 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2023-01-24T17:28:26.627 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-24T17:28:26.628 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T17:28:26.628 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 201463 Links: 1 2023-01-24T17:28:26.628 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T17:28:26.628 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T17:28:26.628 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-01-24 17:28:14.868222716 +0000 2023-01-24T17:28:26.629 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-01-24 17:23:51.166426490 +0000 2023-01-24T17:28:26.629 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-01-24 17:23:51.166426490 +0000 2023-01-24T17:28:26.629 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-01-24T17:28:26.629 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-24T17:28:26.689 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-01-24T17:28:26.689 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-01-24T17:28:26.689 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000221548 s, 2.3 MB/s 2023-01-24T17:28:26.690 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-24T17:28:26.745 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2023-01-24T17:28:26.799 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-24T17:28:26.799 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T17:28:26.800 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 201518 Links: 1 2023-01-24T17:28:26.800 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T17:28:26.800 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T17:28:26.800 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-01-24 17:28:14.868222716 +0000 2023-01-24T17:28:26.800 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-01-24 17:23:51.658416883 +0000 2023-01-24T17:28:26.801 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-01-24 17:23:51.658416883 +0000 2023-01-24T17:28:26.801 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-01-24T17:28:26.801 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-24T17:28:26.861 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-01-24T17:28:26.861 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-01-24T17:28:26.862 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000200822 s, 2.5 MB/s 2023-01-24T17:28:26.863 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-24T17:28:26.917 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T17:28:26.918 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-01-24T17:28:26.918 INFO:tasks.ceph:Generating config... 2023-01-24T17:28:26.919 INFO:tasks.ceph:[client] debug rgw = 20 2023-01-24T17:28:26.920 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-01-24T17:28:26.920 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = vault 2023-01-24T17:28:26.920 INFO:tasks.ceph:[client] rgw crypt sse s3 backend = vault 2023-01-24T17:28:26.920 INFO:tasks.ceph:[client] rgw crypt sse s3 vault auth = token 2023-01-24T17:28:26.921 INFO:tasks.ceph:[client] rgw crypt sse s3 vault prefix = /v1/transit/ 2023-01-24T17:28:26.921 INFO:tasks.ceph:[client] rgw crypt sse s3 vault secret engine = transit 2023-01-24T17:28:26.921 INFO:tasks.ceph:[client] rgw crypt vault auth = token 2023-01-24T17:28:26.921 INFO:tasks.ceph:[client] rgw crypt vault prefix = /v1/transit/ 2023-01-24T17:28:26.922 INFO:tasks.ceph:[client] rgw crypt vault secret engine = transit 2023-01-24T17:28:26.922 INFO:tasks.ceph:[client] setgroup = ceph 2023-01-24T17:28:26.922 INFO:tasks.ceph:[client] setuser = ceph 2023-01-24T17:28:26.922 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-01-24T17:28:26.923 INFO:tasks.ceph:[mgr] debug ms = 1 2023-01-24T17:28:26.923 INFO:tasks.ceph:[mon] debug mon = 20 2023-01-24T17:28:26.924 INFO:tasks.ceph:[mon] debug ms = 1 2023-01-24T17:28:26.924 INFO:tasks.ceph:[mon] debug paxos = 20 2023-01-24T17:28:26.924 INFO:tasks.ceph:[osd] debug ms = 1 2023-01-24T17:28:26.924 INFO:tasks.ceph:[osd] debug osd = 20 2023-01-24T17:28:26.925 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-01-24T17:28:26.925 INFO:tasks.ceph:Setting up mon.a... 2023-01-24T17:28:26.925 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-01-24T17:28:27.019 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.keyring 2023-01-24T17:28:27.021 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-01-24T17:28:27.059 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T17:28:27.122 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.150')] 2023-01-24T17:28:27.122 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.150'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt sse s3 backend': 'vault', 'rgw crypt sse s3 vault auth': 'token', 'rgw crypt sse s3 vault prefix': '/v1/transit/', 'rgw crypt sse s3 vault secret engine': 'transit', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2023-01-24T17:28:27.123 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:27.124 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T17:28:27.176 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.150 --print /home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:28:27.251 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:28:27.251 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:28:27.252 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:28:27.253 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: generated fsid f8f808ab-fc8c-410d-be49-462478484c6d 2023-01-24T17:28:27.253 INFO:teuthology.orchestra.run.smithi150.stdout:setting min_mon_release = octopus 2023-01-24T17:28:27.253 INFO:teuthology.orchestra.run.smithi150.stdout:epoch 0 2023-01-24T17:28:27.253 INFO:teuthology.orchestra.run.smithi150.stdout:fsid f8f808ab-fc8c-410d-be49-462478484c6d 2023-01-24T17:28:27.254 INFO:teuthology.orchestra.run.smithi150.stdout:last_changed 2023-01-24T17:28:27.250217+0000 2023-01-24T17:28:27.254 INFO:teuthology.orchestra.run.smithi150.stdout:created 2023-01-24T17:28:27.250217+0000 2023-01-24T17:28:27.254 INFO:teuthology.orchestra.run.smithi150.stdout:min_mon_release 15 (octopus) 2023-01-24T17:28:27.254 INFO:teuthology.orchestra.run.smithi150.stdout:election_strategy: 1 2023-01-24T17:28:27.254 INFO:teuthology.orchestra.run.smithi150.stdout:0: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.a 2023-01-24T17:28:27.255 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-01-24T17:28:27.255 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T17:28:27.307 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f8f808ab-fc8c-410d-be49-462478484c6d... 2023-01-24T17:28:27.309 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-24T17:28:27.387 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2023-01-24T17:28:27.388 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2023-01-24T17:28:27.388 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-24T17:28:27.388 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2023-01-24T17:28:27.388 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.389 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2023-01-24T17:28:27.389 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.389 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2023-01-24T17:28:27.389 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.389 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2023-01-24T17:28:27.390 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2023-01-24T17:28:27.390 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.390 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2023-01-24T17:28:27.390 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2023-01-24T17:28:27.390 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.391 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-24T17:28:27.391 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2023-01-24T17:28:27.391 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.391 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2023-01-24T17:28:27.391 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-24T17:28:27.392 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.392 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2023-01-24T17:28:27.392 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.392 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2023-01-24T17:28:27.392 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2023-01-24T17:28:27.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2023-01-24T17:28:27.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2023-01-24T17:28:27.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2023-01-24T17:28:27.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2023-01-24T17:28:27.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2023-01-24T17:28:27.394 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-24T17:28:27.394 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-24T17:28:27.394 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2023-01-24T17:28:27.394 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.394 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-24T17:28:27.394 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.395 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2023-01-24T17:28:27.395 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.395 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2023-01-24T17:28:27.395 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.395 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2023-01-24T17:28:27.396 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2023-01-24T17:28:27.396 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2023-01-24T17:28:27.396 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = 172.21.15.150 2023-01-24T17:28:27.396 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = f8f808ab-fc8c-410d-be49-462478484c6d 2023-01-24T17:28:27.396 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.397 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2023-01-24T17:28:27.397 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2023-01-24T17:28:27.397 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.397 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2023-01-24T17:28:27.397 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2023-01-24T17:28:27.398 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.398 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2023-01-24T17:28:27.398 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2023-01-24T17:28:27.398 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.398 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2023-01-24T17:28:27.398 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2023-01-24T17:28:27.399 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2023-01-24T17:28:27.399 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.399 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2023-01-24T17:28:27.399 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2023-01-24T17:28:27.399 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.400 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2023-01-24T17:28:27.400 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.400 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2023-01-24T17:28:27.400 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.400 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2023-01-24T17:28:27.401 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.401 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2023-01-24T17:28:27.401 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2023-01-24T17:28:27.401 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.401 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2023-01-24T17:28:27.402 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2023-01-24T17:28:27.402 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-01-24T17:28:27.402 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2023-01-24T17:28:27.402 INFO:teuthology.orchestra.run.smithi150.stdout: osd shutdown pgref assert = True 2023-01-24T17:28:27.402 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.402 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2023-01-24T17:28:27.403 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-01-24T17:28:27.403 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2023-01-24T17:28:27.403 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2023-01-24T17:28:27.403 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2023-01-24T17:28:27.403 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2023-01-24T17:28:27.404 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2023-01-24T17:28:27.404 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2023-01-24T17:28:27.404 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.404 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2023-01-24T17:28:27.404 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-01-24T17:28:27.405 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2023-01-24T17:28:27.405 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2023-01-24T17:28:27.405 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2023-01-24T17:28:27.405 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2023-01-24T17:28:27.405 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2023-01-24T17:28:27.405 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2023-01-24T17:28:27.406 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2023-01-24T17:28:27.406 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2023-01-24T17:28:27.406 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2023-01-24T17:28:27.406 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.406 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-24T17:28:27.407 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2023-01-24T17:28:27.407 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2023-01-24T17:28:27.407 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.407 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2023-01-24T17:28:27.407 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-24T17:28:27.407 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-24T17:28:27.408 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.408 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2023-01-24T17:28:27.408 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2023-01-24T17:28:27.408 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.409 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2023-01-24T17:28:27.409 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:27.409 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2023-01-24T17:28:27.409 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2023-01-24T17:28:27.409 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2023-01-24T17:28:27.409 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2023-01-24T17:28:27.410 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-24T17:28:27.410 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-24T17:28:27.410 INFO:teuthology.orchestra.run.smithi150.stdout: debug rgw = 20 2023-01-24T17:28:27.410 INFO:teuthology.orchestra.run.smithi150.stdout: rgw crypt require ssl = False 2023-01-24T17:28:27.410 INFO:teuthology.orchestra.run.smithi150.stdout: rgw crypt s3 kms backend = vault 2023-01-24T17:28:27.411 INFO:teuthology.orchestra.run.smithi150.stdout: rgw crypt sse s3 backend = vault 2023-01-24T17:28:27.411 INFO:teuthology.orchestra.run.smithi150.stdout: rgw crypt sse s3 vault auth = token 2023-01-24T17:28:27.411 INFO:teuthology.orchestra.run.smithi150.stdout: rgw crypt sse s3 vault prefix = /v1/transit/ 2023-01-24T17:28:27.411 INFO:teuthology.orchestra.run.smithi150.stdout: rgw crypt sse s3 vault secret engine = transit 2023-01-24T17:28:27.411 INFO:teuthology.orchestra.run.smithi150.stdout: rgw crypt vault auth = token 2023-01-24T17:28:27.412 INFO:teuthology.orchestra.run.smithi150.stdout: rgw crypt vault prefix = /v1/transit/ 2023-01-24T17:28:27.412 INFO:teuthology.orchestra.run.smithi150.stdout: rgw crypt vault secret engine = transit 2023-01-24T17:28:27.412 INFO:teuthology.orchestra.run.smithi150.stdout: setgroup = ceph 2023-01-24T17:28:27.412 INFO:teuthology.orchestra.run.smithi150.stdout: setuser = ceph 2023-01-24T17:28:27.412 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2023-01-24T17:28:27.413 INFO:tasks.ceph:Creating admin key on mon.a... 2023-01-24T17:28:27.414 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-01-24T17:28:27.503 INFO:tasks.ceph:Copying monmap to all nodes... 2023-01-24T17:28:27.503 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:27.504 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-01-24T17:28:27.518 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:27.518 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-01-24T17:28:27.572 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi150.front.sepia.ceph.com 2023-01-24T17:28:27.573 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:27.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-24T17:28:27.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T17:28:27.646 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:27.646 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:28:27.699 INFO:tasks.ceph:Setting up mon nodes... 2023-01-24T17:28:27.700 INFO:tasks.ceph:Setting up mgr nodes... 2023-01-24T17:28:27.700 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T17:28:27.785 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T17:28:27.787 INFO:tasks.ceph:Setting up mds nodes... 2023-01-24T17:28:27.788 INFO:tasks.ceph_client:Setting up client nodes... 2023-01-24T17:28:27.788 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-01-24T17:28:27.825 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-01-24T17:28:27.837 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-01-24T17:28:27.837 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-01-24T17:28:27.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-01-24T17:28:27.901 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T17:28:27.901 INFO:tasks.ceph:role: osd.0 2023-01-24T17:28:27.901 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com 2023-01-24T17:28:27.902 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-24T17:28:27.971 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T17:28:27.972 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T17:28:27.972 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T17:28:27.972 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 2023-01-24T17:28:27.972 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T17:28:27.973 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-01-24T17:28:27.973 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T17:28:27.973 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T17:28:27.973 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T17:28:27.973 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T17:28:28.338 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-01-24T17:28:28.339 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-01-24T17:28:28.340 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-01-24T17:28:28.479 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-01-24T17:28:28.508 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-01-24T17:28:28.571 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T17:28:28.572 INFO:tasks.ceph:role: osd.1 2023-01-24T17:28:28.573 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com 2023-01-24T17:28:28.573 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-24T17:28:28.642 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T17:28:28.643 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T17:28:28.643 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T17:28:28.644 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 2023-01-24T17:28:28.644 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T17:28:28.644 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-01-24T17:28:28.645 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T17:28:28.645 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T17:28:28.645 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T17:28:28.645 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T17:28:29.008 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-01-24T17:28:29.010 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-01-24T17:28:29.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-01-24T17:28:29.095 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-01-24T17:28:29.163 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-01-24T17:28:29.226 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T17:28:29.226 INFO:tasks.ceph:role: osd.2 2023-01-24T17:28:29.226 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com 2023-01-24T17:28:29.227 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-01-24T17:28:29.296 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T17:28:29.297 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T17:28:29.297 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T17:28:29.297 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 2023-01-24T17:28:29.297 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T17:28:29.297 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-01-24T17:28:29.298 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T17:28:29.298 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T17:28:29.298 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T17:28:29.298 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T17:28:29.663 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-01-24T17:28:29.665 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-01-24T17:28:29.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-01-24T17:28:29.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-01-24T17:28:29.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:28:29.863 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:29.860+0000 7f40ffbf3540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-01-24T17:28:29.863 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:29.860+0000 7f40ffbf3540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-01-24T17:28:29.864 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:29.860+0000 7f40ffbf3540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T17:28:29.864 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:29.860+0000 7f40ffbf3540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T17:28:29.864 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:29.860+0000 7f40ffbf3540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T17:28:29.865 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:29.862+0000 7f40ffbf3540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-01-24T17:28:31.944 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-01-24T17:28:32.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:28:32.098 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:32.095+0000 7fcef734a540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-01-24T17:28:32.098 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:32.095+0000 7fcef734a540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-01-24T17:28:32.098 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:32.095+0000 7fcef734a540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T17:28:32.099 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:32.095+0000 7fcef734a540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T17:28:32.099 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:32.095+0000 7fcef734a540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T17:28:32.099 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:32.095+0000 7fcef734a540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-01-24T17:28:34.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-01-24T17:28:34.205 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:28:34.289 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:34.286+0000 7fa656f36540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-01-24T17:28:34.289 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:34.286+0000 7fa656f36540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-01-24T17:28:34.290 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:34.286+0000 7fa656f36540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T17:28:34.290 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:34.286+0000 7fa656f36540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T17:28:34.290 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:34.286+0000 7fa656f36540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T17:28:34.290 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:28:34.287+0000 7fa656f36540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-01-24T17:28:36.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-01-24T17:28:36.436 INFO:tasks.ceph:Reading keys from all nodes... 2023-01-24T17:28:36.437 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:36.437 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-01-24T17:28:36.504 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:36.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-01-24T17:28:36.571 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:36.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-01-24T17:28:36.636 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:36.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-01-24T17:28:36.701 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:36.702 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-01-24T17:28:36.757 INFO:tasks.ceph:Adding keys to all mons... 2023-01-24T17:28:36.757 DEBUG:teuthology.orchestra.run.smithi150:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-24T17:28:36.819 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.x] 2023-01-24T17:28:36.819 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQA7FdBjjf2+LhAAhPdMpx/GM3vaF/KBZ5W9Hw== 2023-01-24T17:28:36.820 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.0] 2023-01-24T17:28:36.820 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQA9FdBjfORcMxAANFdwrEywv48AvfUFnPJgkQ== 2023-01-24T17:28:36.820 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.1] 2023-01-24T17:28:36.820 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQBAFdBj+WS/BRAAbbdw4ODR1h1VrcUtWmM/uQ== 2023-01-24T17:28:36.821 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.2] 2023-01-24T17:28:36.821 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQBCFdBj+LcmERAAU5QTpR1OgzvXMEPZQ62Gkg== 2023-01-24T17:28:36.821 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2023-01-24T17:28:36.821 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQA7FdBj5oIcMRAAZUsEVFZuasi+yXfQiHp44g== 2023-01-24T17:28:36.822 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-24T17:28:36.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T17:28:36.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T17:28:37.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T17:28:37.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-24T17:28:37.096 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-01-24T17:28:37.097 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-01-24T17:28:37.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-24T17:28:37.213 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-01-24T17:28:37.237 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:28:37.292 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-01-24T17:28:37.292 INFO:tasks.ceph.mon.a:Restarting daemon 2023-01-24T17:28:37.293 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-01-24T17:28:37.334 INFO:tasks.ceph.mon.a:Started 2023-01-24T17:28:37.334 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-01-24T17:28:37.335 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-01-24T17:28:37.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-01-24T17:28:37.337 INFO:tasks.ceph.mgr.x:Started 2023-01-24T17:28:37.337 INFO:tasks.ceph:Setting crush tunables to default 2023-01-24T17:28:37.338 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd crush tunables default 2023-01-24T17:28:37.538 INFO:teuthology.orchestra.run.smithi150.stderr:adjusted tunables profile to default 2023-01-24T17:28:37.549 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-01-24T17:28:37.549 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:37.550 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-01-24T17:28:37.573 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:37.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-01-24T17:28:37.639 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:28:37.639 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-01-24T17:28:37.705 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 0b00efc2-6283-4c4f-b400-c4800e5b6679 0 2023-01-24T17:28:37.954 INFO:teuthology.orchestra.run.smithi150.stdout:0 2023-01-24T17:28:37.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new cdbcc5c9-4004-429b-9c41-3098eb189f95 1 2023-01-24T17:28:38.174 INFO:teuthology.orchestra.run.smithi150.stdout:1 2023-01-24T17:28:38.187 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 9156c168-dc0b-4d3c-a01b-15a014386bbd 2 2023-01-24T17:28:38.403 INFO:teuthology.orchestra.run.smithi150.stdout:2 2023-01-24T17:28:38.415 INFO:tasks.ceph.osd.0:Restarting daemon 2023-01-24T17:28:38.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-01-24T17:28:38.418 INFO:tasks.ceph.osd.0:Started 2023-01-24T17:28:38.418 INFO:tasks.ceph.osd.1:Restarting daemon 2023-01-24T17:28:38.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-01-24T17:28:38.421 INFO:tasks.ceph.osd.1:Started 2023-01-24T17:28:38.421 INFO:tasks.ceph.osd.2:Restarting daemon 2023-01-24T17:28:38.422 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-01-24T17:28:38.424 INFO:tasks.ceph.osd.2:Started 2023-01-24T17:28:38.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T17:28:38.637 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:38.637 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":5,"fsid":"f8f808ab-fc8c-410d-be49-462478484c6d","created":"2023-01-24T17:28:37.434400+0000","modified":"2023-01-24T17:28:38.394765+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T17:28:38.394765+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"0b00efc2-6283-4c4f-b400-c4800e5b6679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cdbcc5c9-4004-429b-9c41-3098eb189f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9156c168-dc0b-4d3c-a01b-15a014386bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:28:38.649 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-01-24T17:28:38.649 INFO:tasks.ceph:Waiting for OSDs to come up 2023-01-24T17:28:38.792 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:38.789+0000 7fc958310180 -1 mgr[py] Module not found: 'prometheus' 2023-01-24T17:28:38.793 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:38.789+0000 7fc958310180 -1 mgr[py] Traceback (most recent call last): 2023-01-24T17:28:38.793 INFO:tasks.ceph.mgr.x.smithi150.stderr: File "/usr/share/ceph/mgr/prometheus/__init__.py", line 2, in 2023-01-24T17:28:38.793 INFO:tasks.ceph.mgr.x.smithi150.stderr: from .module import Module, StandbyModule 2023-01-24T17:28:38.793 INFO:tasks.ceph.mgr.x.smithi150.stderr: File "/usr/share/ceph/mgr/prometheus/module.py", line 3, in 2023-01-24T17:28:38.794 INFO:tasks.ceph.mgr.x.smithi150.stderr: from packaging.version import Version 2023-01-24T17:28:38.794 INFO:tasks.ceph.mgr.x.smithi150.stderr:ModuleNotFoundError: No module named 'packaging' 2023-01-24T17:28:38.794 INFO:tasks.ceph.mgr.x.smithi150.stderr: 2023-01-24T17:28:38.795 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:38.790+0000 7fc958310180 -1 mgr[py] Class not found in module 'prometheus' 2023-01-24T17:28:38.795 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:38.790+0000 7fc958310180 -1 mgr[py] Error loading module 'prometheus': (2) No such file or directory 2023-01-24T17:28:38.921 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:38.918+0000 7fc958310180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-24T17:28:38.952 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T17:28:38.997 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:38.994+0000 7fc958310180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-24T17:28:39.019 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-01-24T17:28:39.016+0000 7f03dedd8540 -1 Falling back to public interface 2023-01-24T17:28:39.020 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-01-24T17:28:39.017+0000 7f7fe7dc9540 -1 Falling back to public interface 2023-01-24T17:28:39.028 INFO:tasks.ceph.osd.0.smithi150.stderr:2023-01-24T17:28:39.026+0000 7f59e22dd540 -1 Falling back to public interface 2023-01-24T17:28:39.138 INFO:teuthology.misc.health.smithi150.stdout: 2023-01-24T17:28:39.139 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":5,"fsid":"f8f808ab-fc8c-410d-be49-462478484c6d","created":"2023-01-24T17:28:37.434400+0000","modified":"2023-01-24T17:28:38.394765+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T17:28:38.394765+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"0b00efc2-6283-4c4f-b400-c4800e5b6679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cdbcc5c9-4004-429b-9c41-3098eb189f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9156c168-dc0b-4d3c-a01b-15a014386bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:28:39.149 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-01-24T17:28:39.191 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:39.188+0000 7fc958310180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-24T17:28:39.360 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:39.357+0000 7fc958310180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-24T17:28:39.435 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:39.432+0000 7fc958310180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-24T17:28:39.569 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:39.566+0000 7fc958310180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-24T17:28:39.663 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:39.660+0000 7fc958310180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-24T17:28:40.174 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:40.172+0000 7fc958310180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-24T17:28:40.250 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:40.247+0000 7fc958310180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-24T17:28:40.325 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:40.323+0000 7fc958310180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-24T17:28:40.888 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:40.885+0000 7fc958310180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-01-24T17:28:41.024 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:41.021+0000 7fc958310180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-24T17:28:41.117 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:41.114+0000 7fc958310180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-24T17:28:41.364 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:41.362+0000 7fc958310180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-24T17:28:41.440 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:41.437+0000 7fc958310180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-24T17:28:41.780 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-01-24T17:28:41.778+0000 7f7fe7dc9540 -1 osd.2 0 log_to_monitors true 2023-01-24T17:28:41.789 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-01-24T17:28:41.786+0000 7f03dedd8540 -1 osd.1 0 log_to_monitors true 2023-01-24T17:28:41.803 INFO:tasks.ceph.osd.0.smithi150.stderr:2023-01-24T17:28:41.800+0000 7f59e22dd540 -1 osd.0 0 log_to_monitors true 2023-01-24T17:28:42.020 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:42.018+0000 7fc958310180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-24T17:28:42.259 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:42.256+0000 7fc958310180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-24T17:28:42.386 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:42.384+0000 7fc958310180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-24T17:28:42.468 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:42.466+0000 7fc958310180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-24T17:28:42.614 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:42.611+0000 7fc958310180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-24T17:28:42.687 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:42.685+0000 7fc958310180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-24T17:28:42.759 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:42.756+0000 7fc958310180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-24T17:28:42.832 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:42.829+0000 7fc958310180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-24T17:28:42.832 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:42.829+0000 7fc958310180 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: prometheus 2023-01-24T17:28:44.043 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-01-24T17:28:44.040+0000 7f03d59f9700 -1 osd.1 0 waiting for initial osdmap 2023-01-24T17:28:44.045 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-01-24T17:28:44.042+0000 7f03ce7ef700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T17:28:44.049 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-01-24T17:28:44.047+0000 7f7fde9ea700 -1 osd.2 0 waiting for initial osdmap 2023-01-24T17:28:44.052 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-01-24T17:28:44.049+0000 7fc9117dd700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-01-24T17:28:44.059 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-01-24T17:28:44.057+0000 7f7fd77e0700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T17:28:44.592 INFO:tasks.ceph.osd.0.smithi150.stderr:2023-01-24T17:28:44.589+0000 7f59d8efe700 -1 osd.0 0 waiting for initial osdmap 2023-01-24T17:28:44.594 INFO:tasks.ceph.osd.0.smithi150.stderr:2023-01-24T17:28:44.591+0000 7f59d1cf4700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T17:28:45.452 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T17:28:45.746 INFO:teuthology.misc.health.smithi150.stdout: 2023-01-24T17:28:45.746 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":8,"fsid":"f8f808ab-fc8c-410d-be49-462478484c6d","created":"2023-01-24T17:28:37.434400+0000","modified":"2023-01-24T17:28:45.042153+0000","last_up_change":"2023-01-24T17:28:45.042153+0000","last_in_change":"2023-01-24T17:28:38.394765+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"0b00efc2-6283-4c4f-b400-c4800e5b6679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":79855},{"type":"v1","addr":"172.21.15.150:6817","nonce":79855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":79855},{"type":"v1","addr":"172.21.15.150:6819","nonce":79855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":79855},{"type":"v1","addr":"172.21.15.150:6823","nonce":79855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":79855},{"type":"v1","addr":"172.21.15.150:6821","nonce":79855}]},"public_addr":"172.21.15.150:6817/79855","cluster_addr":"172.21.15.150:6819/79855","heartbeat_back_addr":"172.21.15.150:6823/79855","heartbeat_front_addr":"172.21.15.150:6821/79855","state":["exists","up"]},{"osd":1,"uuid":"cdbcc5c9-4004-429b-9c41-3098eb189f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":79845},{"type":"v1","addr":"172.21.15.150:6801","nonce":79845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":79845},{"type":"v1","addr":"172.21.15.150:6803","nonce":79845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":79845},{"type":"v1","addr":"172.21.15.150:6807","nonce":79845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":79845},{"type":"v1","addr":"172.21.15.150:6805","nonce":79845}]},"public_addr":"172.21.15.150:6801/79845","cluster_addr":"172.21.15.150:6803/79845","heartbeat_back_addr":"172.21.15.150:6807/79845","heartbeat_front_addr":"172.21.15.150:6805/79845","state":["exists","up"]},{"osd":2,"uuid":"9156c168-dc0b-4d3c-a01b-15a014386bbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":79846},{"type":"v1","addr":"172.21.15.150:6809","nonce":79846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":79846},{"type":"v1","addr":"172.21.15.150:6811","nonce":79846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":79846},{"type":"v1","addr":"172.21.15.150:6815","nonce":79846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":79846},{"type":"v1","addr":"172.21.15.150:6813","nonce":79846}]},"public_addr":"172.21.15.150:6809/79846","cluster_addr":"172.21.15.150:6811/79846","heartbeat_back_addr":"172.21.15.150:6815/79846","heartbeat_front_addr":"172.21.15.150:6813/79846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:28:45.758 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-01-24T17:28:45.758 INFO:tasks.ceph:Creating RBD pool 2023-01-24T17:28:45.758 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd pool create rbd 8 2023-01-24T17:28:47.060 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'rbd' created 2023-01-24T17:28:47.073 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-01-24T17:28:48.059 INFO:teuthology.orchestra.run.smithi150.stderr:enabled application 'rbd' on pool 'rbd' 2023-01-24T17:28:48.074 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-01-24T17:28:48.076 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-01-24T17:28:48.077 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-01-24T17:28:48.077 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-01-24T17:28:48.078 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-01-24T17:28:48.428 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:48.440 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":79507},{"type":"v1","addr":"172.21.15.150:6825","nonce":79507}]},"active_addr":"172.21.15.150:6825/79507","active_change":"2023-01-24T17:28:42.858574+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":false,"error_string":"No module named 'packaging'","module_options":{}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":3581696380}]},{"addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":3768144256}]},{"addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":1626247282}]}]}} 2023-01-24T17:28:48.443 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-01-24T17:28:48.443 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-01-24T17:28:48.443 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T17:28:48.741 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:48.741 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":11,"fsid":"f8f808ab-fc8c-410d-be49-462478484c6d","created":"2023-01-24T17:28:37.434400+0000","modified":"2023-01-24T17:28:48.050234+0000","last_up_change":"2023-01-24T17:28:45.042153+0000","last_in_change":"2023-01-24T17:28:38.394765+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T17:28:45.076462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T17:28:46.073389+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0b00efc2-6283-4c4f-b400-c4800e5b6679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":79855},{"type":"v1","addr":"172.21.15.150:6817","nonce":79855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":79855},{"type":"v1","addr":"172.21.15.150:6819","nonce":79855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":79855},{"type":"v1","addr":"172.21.15.150:6823","nonce":79855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":79855},{"type":"v1","addr":"172.21.15.150:6821","nonce":79855}]},"public_addr":"172.21.15.150:6817/79855","cluster_addr":"172.21.15.150:6819/79855","heartbeat_back_addr":"172.21.15.150:6823/79855","heartbeat_front_addr":"172.21.15.150:6821/79855","state":["exists","up"]},{"osd":1,"uuid":"cdbcc5c9-4004-429b-9c41-3098eb189f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":79845},{"type":"v1","addr":"172.21.15.150:6801","nonce":79845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":79845},{"type":"v1","addr":"172.21.15.150:6803","nonce":79845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":79845},{"type":"v1","addr":"172.21.15.150:6807","nonce":79845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":79845},{"type":"v1","addr":"172.21.15.150:6805","nonce":79845}]},"public_addr":"172.21.15.150:6801/79845","cluster_addr":"172.21.15.150:6803/79845","heartbeat_back_addr":"172.21.15.150:6807/79845","heartbeat_front_addr":"172.21.15.150:6805/79845","state":["exists","up"]},{"osd":2,"uuid":"9156c168-dc0b-4d3c-a01b-15a014386bbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":79846},{"type":"v1","addr":"172.21.15.150:6809","nonce":79846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":79846},{"type":"v1","addr":"172.21.15.150:6811","nonce":79846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":79846},{"type":"v1","addr":"172.21.15.150:6815","nonce":79846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":79846},{"type":"v1","addr":"172.21.15.150:6813","nonce":79846}]},"public_addr":"172.21.15.150:6809/79846","cluster_addr":"172.21.15.150:6811/79846","heartbeat_back_addr":"172.21.15.150:6815/79846","heartbeat_front_addr":"172.21.15.150:6813/79846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:28:42.771193+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:28:42.795644+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:28:42.763848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:28:48.755 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-01-24T17:28:48.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T17:28:49.056 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:49.056 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":12,"fsid":"f8f808ab-fc8c-410d-be49-462478484c6d","created":"2023-01-24T17:28:37.434400+0000","modified":"2023-01-24T17:28:49.050878+0000","last_up_change":"2023-01-24T17:28:45.042153+0000","last_in_change":"2023-01-24T17:28:38.394765+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T17:28:45.076462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T17:28:46.073389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0b00efc2-6283-4c4f-b400-c4800e5b6679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":79855},{"type":"v1","addr":"172.21.15.150:6817","nonce":79855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":79855},{"type":"v1","addr":"172.21.15.150:6819","nonce":79855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":79855},{"type":"v1","addr":"172.21.15.150:6823","nonce":79855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":79855},{"type":"v1","addr":"172.21.15.150:6821","nonce":79855}]},"public_addr":"172.21.15.150:6817/79855","cluster_addr":"172.21.15.150:6819/79855","heartbeat_back_addr":"172.21.15.150:6823/79855","heartbeat_front_addr":"172.21.15.150:6821/79855","state":["exists","up"]},{"osd":1,"uuid":"cdbcc5c9-4004-429b-9c41-3098eb189f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":79845},{"type":"v1","addr":"172.21.15.150:6801","nonce":79845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":79845},{"type":"v1","addr":"172.21.15.150:6803","nonce":79845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":79845},{"type":"v1","addr":"172.21.15.150:6807","nonce":79845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":79845},{"type":"v1","addr":"172.21.15.150:6805","nonce":79845}]},"public_addr":"172.21.15.150:6801/79845","cluster_addr":"172.21.15.150:6803/79845","heartbeat_back_addr":"172.21.15.150:6807/79845","heartbeat_front_addr":"172.21.15.150:6805/79845","state":["exists","up"]},{"osd":2,"uuid":"9156c168-dc0b-4d3c-a01b-15a014386bbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":79846},{"type":"v1","addr":"172.21.15.150:6809","nonce":79846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":79846},{"type":"v1","addr":"172.21.15.150:6811","nonce":79846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":79846},{"type":"v1","addr":"172.21.15.150:6815","nonce":79846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":79846},{"type":"v1","addr":"172.21.15.150:6813","nonce":79846}]},"public_addr":"172.21.15.150:6809/79846","cluster_addr":"172.21.15.150:6811/79846","heartbeat_back_addr":"172.21.15.150:6815/79846","heartbeat_front_addr":"172.21.15.150:6813/79846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:28:42.771193+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:28:42.795644+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:28:42.763848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:28:49.072 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-01-24T17:28:49.072 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-01-24T17:28:49.073 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-01-24T17:28:49.228 INFO:teuthology.orchestra.run.smithi150.stdout:34359738371 2023-01-24T17:28:49.229 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T17:28:49.230 INFO:teuthology.orchestra.run.smithi150.stdout:34359738370 2023-01-24T17:28:49.230 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T17:28:49.231 INFO:teuthology.orchestra.run.smithi150.stdout:34359738370 2023-01-24T17:28:49.231 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T17:28:49.584 INFO:teuthology.orchestra.run.smithi150.stdout:34359738369 2023-01-24T17:28:49.598 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2023-01-24T17:28:49.602 INFO:teuthology.orchestra.run.smithi150.stdout:34359738369 2023-01-24T17:28:49.603 INFO:teuthology.orchestra.run.smithi150.stdout:34359738370 2023-01-24T17:28:49.617 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.2 2023-01-24T17:28:49.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2023-01-24T17:28:50.600 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T17:28:50.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T17:28:50.619 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T17:28:50.934 INFO:teuthology.orchestra.run.smithi150.stdout:34359738369 2023-01-24T17:28:50.948 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2023-01-24T17:28:50.980 INFO:teuthology.orchestra.run.smithi150.stdout:34359738370 2023-01-24T17:28:50.989 INFO:teuthology.orchestra.run.smithi150.stdout:34359738369 2023-01-24T17:28:50.994 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2023-01-24T17:28:51.003 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.2 2023-01-24T17:28:51.949 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T17:28:51.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T17:28:52.003 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T17:28:52.291 INFO:teuthology.orchestra.run.smithi150.stdout:34359738371 2023-01-24T17:28:52.305 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738371 for osd.0 2023-01-24T17:28:52.305 DEBUG:teuthology.parallel:result is None 2023-01-24T17:28:52.325 INFO:teuthology.orchestra.run.smithi150.stdout:34359738372 2023-01-24T17:28:52.339 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738372 for osd.1 2023-01-24T17:28:52.339 DEBUG:teuthology.parallel:result is None 2023-01-24T17:28:52.359 INFO:teuthology.orchestra.run.smithi150.stdout:34359738371 2023-01-24T17:28:52.373 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738371 for osd.2 2023-01-24T17:28:52.373 DEBUG:teuthology.parallel:result is None 2023-01-24T17:28:52.373 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-24T17:28:52.374 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T17:28:52.706 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:52.709 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-01-24T17:28:52.719 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-01-24T17:28:51.048370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24912,"kb_used_data":1464,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547888,"statfs":{"total":322122547200,"available":322097037312,"internally_reserved":0,"allocated":1499136,"data_stored":1264483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.060517+0000","last_change":"2023-01-24T17:28:48.057682+0000","last_active":"2023-01-24T17:28:48.060517+0000","last_peered":"2023-01-24T17:28:48.060517+0000","last_clean":"2023-01-24T17:28:48.060517+0000","last_became_active":"2023-01-24T17:28:48.057310+0000","last_became_peered":"2023-01-24T17:28:48.057310+0000","last_unstale":"2023-01-24T17:28:48.060517+0000","last_undegraded":"2023-01-24T17:28:48.060517+0000","last_fullsized":"2023-01-24T17:28:48.060517+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T03:56:50.092484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.060904+0000","last_change":"2023-01-24T17:28:48.057993+0000","last_active":"2023-01-24T17:28:48.060904+0000","last_peered":"2023-01-24T17:28:48.060904+0000","last_clean":"2023-01-24T17:28:48.060904+0000","last_became_active":"2023-01-24T17:28:48.057747+0000","last_became_peered":"2023-01-24T17:28:48.057747+0000","last_unstale":"2023-01-24T17:28:48.060904+0000","last_undegraded":"2023-01-24T17:28:48.060904+0000","last_fullsized":"2023-01-24T17:28:48.060904+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T21:46:17.230406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.060727+0000","last_change":"2023-01-24T17:28:48.057948+0000","last_active":"2023-01-24T17:28:48.060727+0000","last_peered":"2023-01-24T17:28:48.060727+0000","last_clean":"2023-01-24T17:28:48.060727+0000","last_became_active":"2023-01-24T17:28:48.057662+0000","last_became_peered":"2023-01-24T17:28:48.057662+0000","last_unstale":"2023-01-24T17:28:48.060727+0000","last_undegraded":"2023-01-24T17:28:48.060727+0000","last_fullsized":"2023-01-24T17:28:48.060727+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T21:35:32.185001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.066709+0000","last_change":"2023-01-24T17:28:48.061123+0000","last_active":"2023-01-24T17:28:48.066709+0000","last_peered":"2023-01-24T17:28:48.066709+0000","last_clean":"2023-01-24T17:28:48.066709+0000","last_became_active":"2023-01-24T17:28:48.060837+0000","last_became_peered":"2023-01-24T17:28:48.060837+0000","last_unstale":"2023-01-24T17:28:48.066709+0000","last_undegraded":"2023-01-24T17:28:48.066709+0000","last_fullsized":"2023-01-24T17:28:48.066709+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T22:22:42.534627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.052066+0000","last_change":"2023-01-24T17:28:47.059075+0000","last_active":"2023-01-24T17:28:48.052066+0000","last_peered":"2023-01-24T17:28:48.052066+0000","last_clean":"2023-01-24T17:28:48.052066+0000","last_became_active":"2023-01-24T17:28:47.058784+0000","last_became_peered":"2023-01-24T17:28:47.058784+0000","last_unstale":"2023-01-24T17:28:48.052066+0000","last_undegraded":"2023-01-24T17:28:48.052066+0000","last_fullsized":"2023-01-24T17:28:48.052066+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:46.045059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:46.045059+0000","last_clean_scrub_stamp":"2023-01-24T17:28:46.045059+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T01:34:49.564921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.060438+0000","last_change":"2023-01-24T17:28:48.055942+0000","last_active":"2023-01-24T17:28:48.060438+0000","last_peered":"2023-01-24T17:28:48.060438+0000","last_clean":"2023-01-24T17:28:48.060438+0000","last_became_active":"2023-01-24T17:28:48.055692+0000","last_became_peered":"2023-01-24T17:28:48.055692+0000","last_unstale":"2023-01-24T17:28:48.060438+0000","last_undegraded":"2023-01-24T17:28:48.060438+0000","last_fullsized":"2023-01-24T17:28:48.060438+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T23:39:56.723313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.069867+0000","last_change":"2023-01-24T17:28:48.067041+0000","last_active":"2023-01-24T17:28:48.069867+0000","last_peered":"2023-01-24T17:28:48.069867+0000","last_clean":"2023-01-24T17:28:48.069867+0000","last_became_active":"2023-01-24T17:28:48.066657+0000","last_became_peered":"2023-01-24T17:28:48.066657+0000","last_unstale":"2023-01-24T17:28:48.069867+0000","last_undegraded":"2023-01-24T17:28:48.069867+0000","last_fullsized":"2023-01-24T17:28:48.069867+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T03:08:14.641509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.061060+0000","last_change":"2023-01-24T17:28:48.058759+0000","last_active":"2023-01-24T17:28:48.061060+0000","last_peered":"2023-01-24T17:28:48.061060+0000","last_clean":"2023-01-24T17:28:48.061060+0000","last_became_active":"2023-01-24T17:28:48.058504+0000","last_became_peered":"2023-01-24T17:28:48.058504+0000","last_unstale":"2023-01-24T17:28:48.061060+0000","last_undegraded":"2023-01-24T17:28:48.061060+0000","last_fullsized":"2023-01-24T17:28:48.061060+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T21:04:01.197631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.060491+0000","last_change":"2023-01-24T17:28:48.057546+0000","last_active":"2023-01-24T17:28:48.060491+0000","last_peered":"2023-01-24T17:28:48.060491+0000","last_clean":"2023-01-24T17:28:48.060491+0000","last_became_active":"2023-01-24T17:28:48.057304+0000","last_became_peered":"2023-01-24T17:28:48.057304+0000","last_unstale":"2023-01-24T17:28:48.060491+0000","last_undegraded":"2023-01-24T17:28:48.060491+0000","last_fullsized":"2023-01-24T17:28:48.060491+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T20:17:57.691295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7664,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849936,"statfs":{"total":107374182400,"available":107366334464,"internally_reserved":0,"allocated":106496,"data_stored":29071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8620,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848980,"statfs":{"total":107374182400,"available":107365355520,"internally_reserved":0,"allocated":692224,"data_stored":615973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8628,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848972,"statfs":{"total":107374182400,"available":107365347328,"internally_reserved":0,"allocated":700416,"data_stored":619439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T17:28:52.720 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T17:28:53.014 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:53.016 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-01-24T17:28:53.026 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-01-24T17:28:51.048370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24912,"kb_used_data":1464,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547888,"statfs":{"total":322122547200,"available":322097037312,"internally_reserved":0,"allocated":1499136,"data_stored":1264483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.060517+0000","last_change":"2023-01-24T17:28:48.057682+0000","last_active":"2023-01-24T17:28:48.060517+0000","last_peered":"2023-01-24T17:28:48.060517+0000","last_clean":"2023-01-24T17:28:48.060517+0000","last_became_active":"2023-01-24T17:28:48.057310+0000","last_became_peered":"2023-01-24T17:28:48.057310+0000","last_unstale":"2023-01-24T17:28:48.060517+0000","last_undegraded":"2023-01-24T17:28:48.060517+0000","last_fullsized":"2023-01-24T17:28:48.060517+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T03:56:50.092484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.060904+0000","last_change":"2023-01-24T17:28:48.057993+0000","last_active":"2023-01-24T17:28:48.060904+0000","last_peered":"2023-01-24T17:28:48.060904+0000","last_clean":"2023-01-24T17:28:48.060904+0000","last_became_active":"2023-01-24T17:28:48.057747+0000","last_became_peered":"2023-01-24T17:28:48.057747+0000","last_unstale":"2023-01-24T17:28:48.060904+0000","last_undegraded":"2023-01-24T17:28:48.060904+0000","last_fullsized":"2023-01-24T17:28:48.060904+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T21:46:17.230406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.060727+0000","last_change":"2023-01-24T17:28:48.057948+0000","last_active":"2023-01-24T17:28:48.060727+0000","last_peered":"2023-01-24T17:28:48.060727+0000","last_clean":"2023-01-24T17:28:48.060727+0000","last_became_active":"2023-01-24T17:28:48.057662+0000","last_became_peered":"2023-01-24T17:28:48.057662+0000","last_unstale":"2023-01-24T17:28:48.060727+0000","last_undegraded":"2023-01-24T17:28:48.060727+0000","last_fullsized":"2023-01-24T17:28:48.060727+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T21:35:32.185001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.066709+0000","last_change":"2023-01-24T17:28:48.061123+0000","last_active":"2023-01-24T17:28:48.066709+0000","last_peered":"2023-01-24T17:28:48.066709+0000","last_clean":"2023-01-24T17:28:48.066709+0000","last_became_active":"2023-01-24T17:28:48.060837+0000","last_became_peered":"2023-01-24T17:28:48.060837+0000","last_unstale":"2023-01-24T17:28:48.066709+0000","last_undegraded":"2023-01-24T17:28:48.066709+0000","last_fullsized":"2023-01-24T17:28:48.066709+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T22:22:42.534627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.052066+0000","last_change":"2023-01-24T17:28:47.059075+0000","last_active":"2023-01-24T17:28:48.052066+0000","last_peered":"2023-01-24T17:28:48.052066+0000","last_clean":"2023-01-24T17:28:48.052066+0000","last_became_active":"2023-01-24T17:28:47.058784+0000","last_became_peered":"2023-01-24T17:28:47.058784+0000","last_unstale":"2023-01-24T17:28:48.052066+0000","last_undegraded":"2023-01-24T17:28:48.052066+0000","last_fullsized":"2023-01-24T17:28:48.052066+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:46.045059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:46.045059+0000","last_clean_scrub_stamp":"2023-01-24T17:28:46.045059+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T01:34:49.564921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.060438+0000","last_change":"2023-01-24T17:28:48.055942+0000","last_active":"2023-01-24T17:28:48.060438+0000","last_peered":"2023-01-24T17:28:48.060438+0000","last_clean":"2023-01-24T17:28:48.060438+0000","last_became_active":"2023-01-24T17:28:48.055692+0000","last_became_peered":"2023-01-24T17:28:48.055692+0000","last_unstale":"2023-01-24T17:28:48.060438+0000","last_undegraded":"2023-01-24T17:28:48.060438+0000","last_fullsized":"2023-01-24T17:28:48.060438+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T23:39:56.723313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.069867+0000","last_change":"2023-01-24T17:28:48.067041+0000","last_active":"2023-01-24T17:28:48.069867+0000","last_peered":"2023-01-24T17:28:48.069867+0000","last_clean":"2023-01-24T17:28:48.069867+0000","last_became_active":"2023-01-24T17:28:48.066657+0000","last_became_peered":"2023-01-24T17:28:48.066657+0000","last_unstale":"2023-01-24T17:28:48.069867+0000","last_undegraded":"2023-01-24T17:28:48.069867+0000","last_fullsized":"2023-01-24T17:28:48.069867+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T03:08:14.641509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.061060+0000","last_change":"2023-01-24T17:28:48.058759+0000","last_active":"2023-01-24T17:28:48.061060+0000","last_peered":"2023-01-24T17:28:48.061060+0000","last_clean":"2023-01-24T17:28:48.061060+0000","last_became_active":"2023-01-24T17:28:48.058504+0000","last_became_peered":"2023-01-24T17:28:48.058504+0000","last_unstale":"2023-01-24T17:28:48.061060+0000","last_undegraded":"2023-01-24T17:28:48.061060+0000","last_fullsized":"2023-01-24T17:28:48.061060+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T21:04:01.197631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:48.060491+0000","last_change":"2023-01-24T17:28:48.057546+0000","last_active":"2023-01-24T17:28:48.060491+0000","last_peered":"2023-01-24T17:28:48.060491+0000","last_clean":"2023-01-24T17:28:48.060491+0000","last_became_active":"2023-01-24T17:28:48.057304+0000","last_became_peered":"2023-01-24T17:28:48.057304+0000","last_unstale":"2023-01-24T17:28:48.060491+0000","last_undegraded":"2023-01-24T17:28:48.060491+0000","last_fullsized":"2023-01-24T17:28:48.060491+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:47.049406+0000","last_clean_scrub_stamp":"2023-01-24T17:28:47.049406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T20:17:57.691295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7664,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849936,"statfs":{"total":107374182400,"available":107366334464,"internally_reserved":0,"allocated":106496,"data_stored":29071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8620,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848980,"statfs":{"total":107374182400,"available":107365355520,"internally_reserved":0,"allocated":692224,"data_stored":615973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8628,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848972,"statfs":{"total":107374182400,"available":107365347328,"internally_reserved":0,"allocated":700416,"data_stored":619439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T17:28:53.028 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-24T17:28:53.028 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-01-24T17:28:53.028 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-01-24T17:28:53.029 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-01-24T17:28:53.392 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:28:53.393 INFO:teuthology.orchestra.run.smithi150.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-01-24T17:28:53.406 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-01-24T17:28:53.406 INFO:teuthology.run_tasks:Running task vault... 2023-01-24T17:28:53.418 DEBUG:tasks.vault:Vault config is {'client.0': {'engine': 'transit', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/transit/keys/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1'}, {'path': 'my-key-2'}]}} 2023-01-24T17:28:53.419 INFO:tasks.vault:Downloading Vault... 2023-01-24T17:28:53.419 INFO:tasks.vault:Downloading Vault... 2023-01-24T17:28:53.419 DEBUG:teuthology.orchestra.run.smithi150:> curl -L https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip -o /home/ubuntu/cephtest/vault.zip 2023-01-24T17:28:53.458 INFO:teuthology.orchestra.run.smithi150.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:28:53.459 INFO:teuthology.orchestra.run.smithi150.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:28:53.649 INFO:teuthology.orchestra.run.smithi150.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 81 42.6M 81 34.8M 0 0 204M 0 --:--:-- --:--:-- --:--:-- 203M 100 42.6M 100 42.6M 0 0 223M 0 --:--:-- --:--:-- --:--:-- 222M 2023-01-24T17:28:53.650 INFO:tasks.vault:Verifying SHA256 signature... 2023-01-24T17:28:53.651 DEBUG:teuthology.orchestra.run.smithi150:> echo '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 /home/ubuntu/cephtest/vault.zip' | sha256sum --check --status 2023-01-24T17:28:53.820 INFO:tasks.vault:Extracting vault... 2023-01-24T17:28:53.821 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p /home/ubuntu/cephtest/vault 2023-01-24T17:28:53.836 DEBUG:teuthology.orchestra.run.smithi150:> unzip /home/ubuntu/cephtest/vault.zip -d /home/ubuntu/cephtest/vault 2023-01-24T17:28:53.897 INFO:teuthology.orchestra.run.smithi150.stdout:Archive: /home/ubuntu/cephtest/vault.zip 2023-01-24T17:28:54.614 INFO:teuthology.orchestra.run.smithi150.stdout: inflating: /home/ubuntu/cephtest/vault/vault 2023-01-24T17:28:54.615 INFO:tasks.vault:Starting Vault listening on 0.0.0.0:8200 ... 2023-01-24T17:28:54.615 INFO:tasks.vault.client.0:Restarting daemon 2023-01-24T17:28:54.616 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/vault && exec bash -c 'chmod +x /home/ubuntu/cephtest/vault/vault && /home/ubuntu/cephtest/vault/vault server -dev -dev-listen-address=0.0.0.0:8200 -dev-no-store-token -dev-root-token-id=test_root_token' & { read; kill %1; }) 2023-01-24T17:28:54.657 INFO:tasks.vault.client.0:Started 2023-01-24T17:28:54.700 INFO:tasks.vault.client.0.smithi150.stdout:==> Vault server configuration: 2023-01-24T17:28:54.701 INFO:tasks.vault.client.0.smithi150.stdout: 2023-01-24T17:28:54.701 INFO:tasks.vault.client.0.smithi150.stdout: Api Address: http://0.0.0.0:8200 2023-01-24T17:28:54.701 INFO:tasks.vault.client.0.smithi150.stdout: Cgo: disabled 2023-01-24T17:28:54.701 INFO:tasks.vault.client.0.smithi150.stdout: Cluster Address: https://0.0.0.0:8201 2023-01-24T17:28:54.701 INFO:tasks.vault.client.0.smithi150.stdout: Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled") 2023-01-24T17:28:54.702 INFO:tasks.vault.client.0.smithi150.stdout: Log Level: info 2023-01-24T17:28:54.702 INFO:tasks.vault.client.0.smithi150.stdout: Mlock: supported: true, enabled: false 2023-01-24T17:28:54.702 INFO:tasks.vault.client.0.smithi150.stdout: Storage: inmem 2023-01-24T17:28:54.702 INFO:tasks.vault.client.0.smithi150.stdout: Version: Vault v1.2.2 2023-01-24T17:28:54.702 INFO:tasks.vault.client.0.smithi150.stdout: 2023-01-24T17:28:54.717 INFO:tasks.vault.client.0.smithi150.stdout:WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory 2023-01-24T17:28:54.717 INFO:tasks.vault.client.0.smithi150.stdout:and starts unsealed with a single unseal key. The root token is already 2023-01-24T17:28:54.717 INFO:tasks.vault.client.0.smithi150.stdout:authenticated to the CLI, so you can immediately begin using Vault. 2023-01-24T17:28:54.718 INFO:tasks.vault.client.0.smithi150.stdout: 2023-01-24T17:28:54.718 INFO:tasks.vault.client.0.smithi150.stdout:You may need to set the following environment variable: 2023-01-24T17:28:54.718 INFO:tasks.vault.client.0.smithi150.stdout: 2023-01-24T17:28:54.718 INFO:tasks.vault.client.0.smithi150.stdout: $ export VAULT_ADDR='http://0.0.0.0:8200' 2023-01-24T17:28:54.719 INFO:tasks.vault.client.0.smithi150.stdout: 2023-01-24T17:28:54.719 INFO:tasks.vault.client.0.smithi150.stdout:The unseal key and root token are displayed below in case you want to 2023-01-24T17:28:54.719 INFO:tasks.vault.client.0.smithi150.stdout:seal/unseal the Vault or re-authenticate. 2023-01-24T17:28:54.719 INFO:tasks.vault.client.0.smithi150.stdout: 2023-01-24T17:28:54.719 INFO:tasks.vault.client.0.smithi150.stdout:Unseal Key: mJ8uK14oXLq/l2HARW1HFxhpWFiPFGTU1bzp6f69EpY= 2023-01-24T17:28:54.720 INFO:tasks.vault.client.0.smithi150.stdout:Root Token: test_root_token 2023-01-24T17:28:54.720 INFO:tasks.vault.client.0.smithi150.stdout: 2023-01-24T17:28:54.720 INFO:tasks.vault.client.0.smithi150.stdout:Development mode should NOT be used in production installations! 2023-01-24T17:28:54.721 INFO:tasks.vault.client.0.smithi150.stdout: 2023-01-24T17:28:54.721 INFO:tasks.vault.client.0.smithi150.stdout:==> Vault server started! Log data will stream in below: 2023-01-24T17:28:54.721 INFO:tasks.vault.client.0.smithi150.stdout: 2023-01-24T17:28:54.722 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.693Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set 2023-01-24T17:28:54.723 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.694Z [INFO] core: security barrier not initialized 2023-01-24T17:28:54.723 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.694Z [INFO] core: security barrier initialized: shares=1 threshold=1 2023-01-24T17:28:54.724 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.694Z [INFO] core: post-unseal setup starting 2023-01-24T17:28:54.724 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.702Z [INFO] core: loaded wrapping token key 2023-01-24T17:28:54.724 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.702Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2023-01-24T17:28:54.725 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.702Z [INFO] core: no mounts; adding default mount table 2023-01-24T17:28:54.725 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.703Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2023-01-24T17:28:54.725 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.704Z [INFO] core: successfully mounted backend: type=system path=sys/ 2023-01-24T17:28:54.726 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.704Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2023-01-24T17:28:54.726 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.705Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2023-01-24T17:28:54.726 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.705Z [INFO] core: restoring leases 2023-01-24T17:28:54.727 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.705Z [INFO] rollback: starting rollback manager 2023-01-24T17:28:54.727 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.705Z [INFO] expiration: lease restore complete 2023-01-24T17:28:54.727 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.706Z [INFO] identity: entities restored 2023-01-24T17:28:54.727 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.706Z [INFO] identity: groups restored 2023-01-24T17:28:54.728 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.706Z [INFO] core: post-unseal setup complete 2023-01-24T17:28:54.728 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.706Z [INFO] core: root token generated 2023-01-24T17:28:54.728 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.706Z [INFO] core: pre-seal teardown starting 2023-01-24T17:28:54.729 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.706Z [INFO] rollback: stopping rollback manager 2023-01-24T17:28:54.729 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.706Z [INFO] core: pre-seal teardown complete 2023-01-24T17:28:54.729 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.706Z [INFO] core.cluster-listener: starting listener: listener_address=0.0.0.0:8201 2023-01-24T17:28:54.729 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.707Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201 2023-01-24T17:28:54.729 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.707Z [INFO] core: post-unseal setup starting 2023-01-24T17:28:54.730 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.707Z [INFO] core: loaded wrapping token key 2023-01-24T17:28:54.730 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.707Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2023-01-24T17:28:54.730 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.707Z [INFO] core: successfully mounted backend: type=system path=sys/ 2023-01-24T17:28:54.730 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.707Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2023-01-24T17:28:54.730 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.707Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2023-01-24T17:28:54.731 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.707Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2023-01-24T17:28:54.731 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.708Z [INFO] core: restoring leases 2023-01-24T17:28:54.731 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.708Z [INFO] rollback: starting rollback manager 2023-01-24T17:28:54.731 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.708Z [INFO] identity: entities restored 2023-01-24T17:28:54.731 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.708Z [INFO] identity: groups restored 2023-01-24T17:28:54.731 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.708Z [INFO] core: post-unseal setup complete 2023-01-24T17:28:54.732 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.708Z [INFO] core: vault is unsealed 2023-01-24T17:28:54.732 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.708Z [INFO] expiration: lease restore complete 2023-01-24T17:28:54.732 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.709Z [INFO] expiration: revoked lease: lease_id=auth/token/root/he00fabb1aa49f15e89ec79e771443a754e4ffeb12ebd0e8e0fef931bdf70646b 2023-01-24T17:28:54.732 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.710Z [INFO] core: successful mount: namespace= path=secret/ type=kv 2023-01-24T17:28:54.733 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.720Z [INFO] secrets.kv.kv_e179431d: collecting keys to upgrade 2023-01-24T17:28:54.733 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.720Z [INFO] secrets.kv.kv_e179431d: done collecting keys: num_keys=1 2023-01-24T17:28:54.733 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:28:54.720Z [INFO] secrets.kv.kv_e179431d: upgrading keys finished 2023-01-24T17:29:04.658 INFO:tasks.vault:Mounting transit secrets engine 2023-01-24T17:29:04.659 INFO:tasks.vault:Send request to Vault: smithi150.front.sepia.ceph.com:8200 at /v1/sys/mounts/transit with token: test_root_token 2023-01-24T17:29:04.664 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:29:04.663Z [INFO] core: successful mount: namespace= path=transit/ type=transit 2023-01-24T17:29:04.665 INFO:tasks.vault:b'' 2023-01-24T17:29:04.665 INFO:tasks.vault:Send request to Vault: smithi150.front.sepia.ceph.com:8200 at /v1/transit/keys/my-key-1 with token: test_root_token 2023-01-24T17:29:04.667 INFO:tasks.vault:b'' 2023-01-24T17:29:04.668 INFO:tasks.vault:Send request to Vault: smithi150.front.sepia.ceph.com:8200 at /v1/transit/keys/my-key-2 with token: test_root_token 2023-01-24T17:29:04.670 INFO:tasks.vault:b'' 2023-01-24T17:29:04.670 INFO:tasks.vault:secrets created 2023-01-24T17:29:04.670 INFO:teuthology.run_tasks:Running task rgw... 2023-01-24T17:29:04.685 DEBUG:tasks.rgw:config is {'client.0': {'use-vault-role': 'client.0'}} 2023-01-24T17:29:04.685 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-01-24T17:29:04.686 INFO:tasks.rgw:Creating data pools 2023-01-24T17:29:04.686 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-01-24T17:29:04.686 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2023-01-24T17:29:05.434 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'default.rgw.buckets.data' created 2023-01-24T17:29:05.449 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-01-24T17:29:06.446 INFO:teuthology.orchestra.run.smithi150.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-01-24T17:29:06.464 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-01-24T17:29:07.463 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'default.rgw.buckets.index' created 2023-01-24T17:29:07.475 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-01-24T17:29:08.456 INFO:teuthology.orchestra.run.smithi150.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-01-24T17:29:08.472 DEBUG:tasks.rgw:Pools created 2023-01-24T17:29:08.473 INFO:tasks.rgw:Starting rgw... 2023-01-24T17:29:08.473 INFO:tasks.rgw:rgw client.0 config is {'use-vault-role': 'client.0'} 2023-01-24T17:29:08.473 INFO:tasks.rgw:Using beast as radosgw frontend 2023-01-24T17:29:08.474 DEBUG:teuthology.orchestra.run.smithi150:> sudo echo -n test_root_token | sudo tee /etc/ceph/vault-root-token 2023-01-24T17:29:08.502 INFO:teuthology.orchestra.run.smithi150.stdout:test_root_token 2023-01-24T17:29:08.502 INFO:tasks.rgw:Token file content 2023-01-24T17:29:08.503 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/ceph/vault-root-token 2023-01-24T17:29:08.556 INFO:teuthology.orchestra.run.smithi150.stdout:test_root_token 2023-01-24T17:29:08.557 INFO:tasks.rgw:Restrict access to token file 2023-01-24T17:29:08.557 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 600 /etc/ceph/vault-root-token 2023-01-24T17:29:08.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown ceph /etc/ceph/vault-root-token 2023-01-24T17:29:08.683 INFO:tasks.rgw.client.0:Restarting daemon 2023-01-24T17:29:08.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi150.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --rgw_crypt_sse_s3_vault_addr smithi150.front.sepia.ceph.com:8200 --rgw_crypt_sse_s3_vault_token_file /etc/ceph/vault-root-token --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-01-24T17:29:08.726 INFO:tasks.rgw.client.0:Started 2023-01-24T17:29:08.726 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi150.front.sepia.ceph.com:80/ 2023-01-24T17:29:08.727 DEBUG:teuthology.orchestra.run.smithi150:> curl http://smithi150.front.sepia.ceph.com:80/ 2023-01-24T17:29:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T17:29:08.744 INFO:teuthology.orchestra.run.smithi150.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:29:08.745 INFO:teuthology.orchestra.run.smithi150.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:29:08.745 INFO:teuthology.orchestra.run.smithi150.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi150.front.sepia.ceph.com port 80: Connection refused 2023-01-24T17:29:09.746 DEBUG:teuthology.orchestra.run.smithi150:> curl http://smithi150.front.sepia.ceph.com:80/ 2023-01-24T17:29:09.765 INFO:teuthology.orchestra.run.smithi150.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:29:09.766 INFO:teuthology.orchestra.run.smithi150.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:29:09.766 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T17:29:09.767 INFO:teuthology.orchestra.run.smithi150.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi150.front.sepia.ceph.com port 80: Connection refused 2023-01-24T17:29:11.769 DEBUG:teuthology.orchestra.run.smithi150:> curl http://smithi150.front.sepia.ceph.com:80/ 2023-01-24T17:29:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T17:29:11.792 INFO:teuthology.orchestra.run.smithi150.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:29:11.792 INFO:teuthology.orchestra.run.smithi150.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:29:11.792 INFO:teuthology.orchestra.run.smithi150.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi150.front.sepia.ceph.com port 80: Connection refused 2023-01-24T17:29:15.793 DEBUG:teuthology.orchestra.run.smithi150:> curl http://smithi150.front.sepia.ceph.com:80/ 2023-01-24T17:29:15.816 INFO:teuthology.orchestra.run.smithi150.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:29:15.816 INFO:teuthology.orchestra.run.smithi150.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:29:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T17:29:15.818 INFO:teuthology.orchestra.run.smithi150.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi150.front.sepia.ceph.com port 80: Connection refused 2023-01-24T17:29:23.819 DEBUG:teuthology.orchestra.run.smithi150:> curl http://smithi150.front.sepia.ceph.com:80/ 2023-01-24T17:29:23.841 INFO:teuthology.orchestra.run.smithi150.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:29:23.841 INFO:teuthology.orchestra.run.smithi150.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:29:23.842 INFO:teuthology.orchestra.run.smithi150.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2023-01-24T17:29:23.843 INFO:teuthology.orchestra.run.smithi150.stdout:anonymous 2023-01-24T17:29:23.843 INFO:teuthology.run_tasks:Running task s3tests... 2023-01-24T17:29:23.863 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'with-sse-s3': True}} 2023-01-24T17:29:23.863 INFO:tasks.s3tests:Downloading s3-tests... 2023-01-24T17:29:23.863 INFO:tasks.s3tests:Using branch 'ceph-quincy' for s3tests 2023-01-24T17:29:23.864 DEBUG:teuthology.orchestra.run.smithi150:> git clone -b ceph-quincy https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests 2023-01-24T17:29:23.900 INFO:teuthology.orchestra.run.smithi150.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests'... 2023-01-24T17:29:25.250 INFO:tasks.s3tests:Creating rgw users... 2023-01-24T17:29:25.250 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2023-01-24T17:29:25.251 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key AMQVQYCNKWIFDXUYMOIN --secret tXaz4+2pMk/Hl+D7FVIBOrbCTtwYwrZ3FvkZcbgiLw1DjlwX0mh0Bw== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-01-24T17:29:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:29:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:29:25.351 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.349+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.350+0000 7f7928d3d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.350+0000 7f7928d3d680 20 realm 2023-01-24T17:29:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.350+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.350+0000 7f7928d3d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.350+0000 7f7928d3d680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:29:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.350+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.350+0000 7f7928d3d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.355 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.350+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.355 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.351+0000 7f7928d3d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:25.355 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.351+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.356 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.351+0000 7f7928d3d680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:29:25.356 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.351+0000 7f7928d3d680 20 searching for the correct realm 2023-01-24T17:29:25.362 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.359+0000 7f7928d3d680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:29:25.362 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.359+0000 7f7928d3d680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:29:25.363 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.359+0000 7f7928d3d680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:29:25.363 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.360+0000 7f7928d3d680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:29:25.363 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.360+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.363 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.360+0000 7f7928d3d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.364 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.360+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.365 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.360+0000 7f7928d3d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:25.365 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.360+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.365 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.360+0000 7f7928d3d680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:29:25.365 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.360+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.360+0000 7f7928d3d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.360+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.361+0000 7f7928d3d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.361+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.361+0000 7f7928d3d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:25.371 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.368+0000 7f7928d3d680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:29:25.371 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.368+0000 7f7928d3d680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:29:25.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.368+0000 7f7928d3d680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:29:25.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.368+0000 7f7928d3d680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:29:25.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.368+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.368+0000 7f7928d3d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:25.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.368+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.368+0000 7f7928d3d680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:29:25.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.368+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.369+0000 7f7928d3d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:25.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.369+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.369+0000 7f7928d3d680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:29:25.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.369+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.369+0000 7f7928d3d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.369+0000 7f7928d3d680 20 zone default found 2023-01-24T17:29:25.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.369+0000 7f7928d3d680 4 Realm: () 2023-01-24T17:29:25.376 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.369+0000 7f7928d3d680 4 ZoneGroup: default (ea8e6aae-de17-4da7-a2fe-dcef04325757) 2023-01-24T17:29:25.376 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.369+0000 7f7928d3d680 4 Zone: default (60550c17-6c7c-4ff3-b915-efe39561b124) 2023-01-24T17:29:25.376 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.369+0000 7f7928d3d680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:29:25.376 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.369+0000 7f7928d3d680 20 zonegroup default 2023-01-24T17:29:25.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.369+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.370+0000 7f7928d3d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.370+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.370+0000 7f7928d3d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.370+0000 7f7928d3d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.370+0000 7f7928d3d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.378 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.370+0000 7f7928d3d680 20 started sync module instance, tier type = 2023-01-24T17:29:25.378 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.370+0000 7f7928d3d680 20 started zone id=60550c17-6c7c-4ff3-b915-efe39561b124 (name=default) with tier type = 2023-01-24T17:29:25.391 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.388+0000 7f7928d3d680 20 add_watcher() i=0 2023-01-24T17:29:25.391 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.388+0000 7f7928d3d680 20 add_watcher() i=1 2023-01-24T17:29:25.391 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.388+0000 7f7928d3d680 20 add_watcher() i=2 2023-01-24T17:29:25.392 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.388+0000 7f7928d3d680 20 add_watcher() i=3 2023-01-24T17:29:25.392 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.388+0000 7f7928d3d680 20 add_watcher() i=4 2023-01-24T17:29:25.392 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.388+0000 7f7928d3d680 20 add_watcher() i=5 2023-01-24T17:29:25.392 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.388+0000 7f7928d3d680 20 add_watcher() i=6 2023-01-24T17:29:25.393 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.389+0000 7f7928d3d680 20 add_watcher() i=7 2023-01-24T17:29:25.393 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.389+0000 7f7928d3d680 2 all 8 watchers are set, enabling cache 2023-01-24T17:29:25.393 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.391+0000 7f78ed7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:29:25.394 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.391+0000 7f7928d3d680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:29:25.394 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.391+0000 7f7928d3d680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:29:25.397 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.395+0000 7f7928d3d680 5 note: GC not initialized 2023-01-24T17:29:25.400 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.396+0000 7f785bfff700 20 reqs_thread_entry: start 2023-01-24T17:29:25.400 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.396+0000 7f7859ffb700 20 reqs_thread_entry: start 2023-01-24T17:29:25.400 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.396+0000 7f7928d3d680 20 init_complete bucket index max shards: 11 2023-01-24T17:29:25.400 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.396+0000 7f7928d3d680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:29:25.400 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.396+0000 7f7847fff700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:29:55 2023 2023-01-24T17:29:25.401 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-01-24T17:29:25.401 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.396+0000 7f78477fe700 20 reqs_thread_entry: start 2023-01-24T17:29:25.401 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.396+0000 7f7928d3d680 20 get_system_obj_state: rctx=0x7fff92e22088 obj=default.rgw.meta:users.uid:foo.client.0 state=0x556f55422940 s->prefetch_data=0 2023-01-24T17:29:25.401 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.396+0000 7f7928d3d680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-01-24T17:29:25.401 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.396+0000 7f7928d3d680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2023-01-24T17:29:25.402 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.396+0000 7f7928d3d680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:29:25.402 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.396+0000 7f7928d3d680 20 get_system_obj_state: rctx=0x7fff92e21d78 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x556f55422940 s->prefetch_data=0 2023-01-24T17:29:25.402 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.396+0000 7f7928d3d680 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2023-01-24T17:29:25.402 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.397+0000 7f7928d3d680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2023-01-24T17:29:25.402 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.397+0000 7f7928d3d680 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-01-24T17:29:25.403 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.397+0000 7f7928d3d680 20 get_system_obj_state: rctx=0x7fff92e21d78 obj=default.rgw.meta:users.keys:AMQVQYCNKWIFDXUYMOIN state=0x556f55422940 s->prefetch_data=0 2023-01-24T17:29:25.403 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.397+0000 7f7928d3d680 10 cache get: name=default.rgw.meta+users.keys+AMQVQYCNKWIFDXUYMOIN : miss 2023-01-24T17:29:25.403 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.397+0000 7f7928d3d680 10 cache put: name=default.rgw.meta+users.keys+AMQVQYCNKWIFDXUYMOIN info.flags=0x0 2023-01-24T17:29:25.403 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.397+0000 7f7928d3d680 10 adding default.rgw.meta+users.keys+AMQVQYCNKWIFDXUYMOIN to cache LRU end 2023-01-24T17:29:25.403 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.397+0000 7f7928d3d680 20 get_system_obj_state: rctx=0x7fff92e21608 obj=default.rgw.meta:users.keys:AMQVQYCNKWIFDXUYMOIN state=0x556f55422940 s->prefetch_data=0 2023-01-24T17:29:25.404 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.397+0000 7f7928d3d680 10 cache get: name=default.rgw.meta+users.keys+AMQVQYCNKWIFDXUYMOIN : hit (negative entry) 2023-01-24T17:29:25.404 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.397+0000 7f7928d3d680 20 get_system_obj_state: rctx=0x7fff92e21cb8 obj=default.rgw.meta:users.keys:AMQVQYCNKWIFDXUYMOIN state=0x556f55422940 s->prefetch_data=0 2023-01-24T17:29:25.404 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.397+0000 7f7928d3d680 10 cache get: name=default.rgw.meta+users.keys+AMQVQYCNKWIFDXUYMOIN : hit (negative entry) 2023-01-24T17:29:25.405 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.398+0000 7f7928d3d680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T17:29:25.405 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.398+0000 7f7928d3d680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:29:25.406 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.398+0000 7f7928d3d680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-01-24T17:29:25.406 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.398+0000 7f791a63c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 93936661719008 notifier 4219 bl.length()=640 2023-01-24T17:29:25.406 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.398+0000 7f791a63c700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T17:29:25.406 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.398+0000 7f791a63c700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:29:25.406 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.401+0000 7f7928d3d680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-01-24T17:29:25.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.401+0000 7f7928d3d680 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-01-24T17:29:25.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.401+0000 7f7928d3d680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-01-24T17:29:25.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.401+0000 7f791a63c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 93936661754992 notifier 4219 bl.length()=194 2023-01-24T17:29:25.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.401+0000 7f791a63c700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-01-24T17:29:25.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.401+0000 7f791a63c700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-01-24T17:29:25.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.403+0000 7f7928d3d680 10 cache put: name=default.rgw.meta+users.keys+AMQVQYCNKWIFDXUYMOIN info.flags=0x7 2023-01-24T17:29:25.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.403+0000 7f7928d3d680 10 moving default.rgw.meta+users.keys+AMQVQYCNKWIFDXUYMOIN to cache LRU end 2023-01-24T17:29:25.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.403+0000 7f7928d3d680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.keys:AMQVQYCNKWIFDXUYMOIN, ofs0, ns] 2023-01-24T17:29:25.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.404+0000 7f7919e3b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182401 cookie 93936661759472 notifier 4219 bl.length()=186 2023-01-24T17:29:25.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.404+0000 7f7919e3b700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+AMQVQYCNKWIFDXUYMOIN info.flags=0x7 2023-01-24T17:29:25.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.404+0000 7f7919e3b700 10 rgw watcher librados: moving default.rgw.meta+users.keys+AMQVQYCNKWIFDXUYMOIN to cache LRU end 2023-01-24T17:29:25.410 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-01-24T17:29:25.410 INFO:teuthology.orchestra.run.smithi150.stdout: "user_id": "foo.client.0", 2023-01-24T17:29:25.410 INFO:teuthology.orchestra.run.smithi150.stdout: "display_name": "Mr. foo.client.0", 2023-01-24T17:29:25.410 INFO:teuthology.orchestra.run.smithi150.stdout: "email": "foo.client.0+test@test.test", 2023-01-24T17:29:25.410 INFO:teuthology.orchestra.run.smithi150.stdout: "suspended": 0, 2023-01-24T17:29:25.411 INFO:teuthology.orchestra.run.smithi150.stdout: "max_buckets": 1000, 2023-01-24T17:29:25.411 INFO:teuthology.orchestra.run.smithi150.stdout: "subusers": [], 2023-01-24T17:29:25.411 INFO:teuthology.orchestra.run.smithi150.stdout: "keys": [ 2023-01-24T17:29:25.411 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-01-24T17:29:25.411 INFO:teuthology.orchestra.run.smithi150.stdout: "user": "foo.client.0", 2023-01-24T17:29:25.412 INFO:teuthology.orchestra.run.smithi150.stdout: "access_key": "AMQVQYCNKWIFDXUYMOIN", 2023-01-24T17:29:25.412 INFO:teuthology.orchestra.run.smithi150.stdout: "secret_key": "tXaz4+2pMk/Hl+D7FVIBOrbCTtwYwrZ3FvkZcbgiLw1DjlwX0mh0Bw==" 2023-01-24T17:29:25.412 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-01-24T17:29:25.412 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2023-01-24T17:29:25.412 INFO:teuthology.orchestra.run.smithi150.stdout: "swift_keys": [], 2023-01-24T17:29:25.413 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": [ 2023-01-24T17:29:25.413 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-01-24T17:29:25.413 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "user-policy", 2023-01-24T17:29:25.413 INFO:teuthology.orchestra.run.smithi150.stdout: "perm": "*" 2023-01-24T17:29:25.413 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-01-24T17:29:25.414 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2023-01-24T17:29:25.414 INFO:teuthology.orchestra.run.smithi150.stdout: "op_mask": "read, write, delete", 2023-01-24T17:29:25.414 INFO:teuthology.orchestra.run.smithi150.stdout: "default_placement": "", 2023-01-24T17:29:25.414 INFO:teuthology.orchestra.run.smithi150.stdout: "default_storage_class": "", 2023-01-24T17:29:25.414 INFO:teuthology.orchestra.run.smithi150.stdout: "placement_tags": [], 2023-01-24T17:29:25.415 INFO:teuthology.orchestra.run.smithi150.stdout: "bucket_quota": { 2023-01-24T17:29:25.415 INFO:teuthology.orchestra.run.smithi150.stdout: "enabled": false, 2023-01-24T17:29:25.415 INFO:teuthology.orchestra.run.smithi150.stdout: "check_on_raw": false, 2023-01-24T17:29:25.415 INFO:teuthology.orchestra.run.smithi150.stdout: "max_size": -1, 2023-01-24T17:29:25.415 INFO:teuthology.orchestra.run.smithi150.stdout: "max_size_kb": 0, 2023-01-24T17:29:25.416 INFO:teuthology.orchestra.run.smithi150.stdout: "max_objects": -1 2023-01-24T17:29:25.416 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-01-24T17:29:25.416 INFO:teuthology.orchestra.run.smithi150.stdout: "user_quota": { 2023-01-24T17:29:25.416 INFO:teuthology.orchestra.run.smithi150.stdout: "enabled": false, 2023-01-24T17:29:25.416 INFO:teuthology.orchestra.run.smithi150.stdout: "check_on_raw": false, 2023-01-24T17:29:25.416 INFO:teuthology.orchestra.run.smithi150.stdout: "max_size": -1, 2023-01-24T17:29:25.417 INFO:teuthology.orchestra.run.smithi150.stdout: "max_size_kb": 0, 2023-01-24T17:29:25.417 INFO:teuthology.orchestra.run.smithi150.stdout: "max_objects": -1 2023-01-24T17:29:25.417 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-01-24T17:29:25.417 INFO:teuthology.orchestra.run.smithi150.stdout: "temp_url_keys": [], 2023-01-24T17:29:25.417 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "rgw", 2023-01-24T17:29:25.418 INFO:teuthology.orchestra.run.smithi150.stdout: "mfa_ids": [] 2023-01-24T17:29:25.418 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-01-24T17:29:25.418 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:29:25.420 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.405+0000 7f7928d3d680 20 remove_watcher() i=0 2023-01-24T17:29:25.420 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.405+0000 7f7928d3d680 2 removed watcher, disabling cache 2023-01-24T17:29:25.420 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.406+0000 7f7928d3d680 20 remove_watcher() i=1 2023-01-24T17:29:25.420 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.407+0000 7f7928d3d680 20 remove_watcher() i=2 2023-01-24T17:29:25.421 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.409+0000 7f7928d3d680 20 remove_watcher() i=3 2023-01-24T17:29:25.421 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.410+0000 7f7928d3d680 20 remove_watcher() i=4 2023-01-24T17:29:25.421 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.410+0000 7f7928d3d680 20 remove_watcher() i=5 2023-01-24T17:29:25.421 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.411+0000 7f7928d3d680 20 remove_watcher() i=6 2023-01-24T17:29:25.421 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.412+0000 7f7928d3d680 20 remove_watcher() i=7 2023-01-24T17:29:25.422 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.413+0000 7f78ed7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:29:25.422 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 1507549550 --totp-seed E7S6TUBICCOGHL4JMOBO7YHZKQAFU2JPSEGC2ZYKQ2FNKQE5UT726COM7UHBNFKI --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-01-24T17:29:25.470 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:29:25.470 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:29:25.475 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.473+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.477 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.474+0000 7fa085223680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.477 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.474+0000 7fa085223680 20 realm 2023-01-24T17:29:25.477 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.474+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.478 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.474+0000 7fa085223680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.478 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.474+0000 7fa085223680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:29:25.478 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.474+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.478 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.474+0000 7fa085223680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.478 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.474+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.479 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.474+0000 7fa085223680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:25.479 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.474+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.479 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.475+0000 7fa085223680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:29:25.480 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.475+0000 7fa085223680 20 searching for the correct realm 2023-01-24T17:29:25.485 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.482+0000 7fa085223680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:29:25.485 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.482+0000 7fa085223680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:29:25.485 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.482+0000 7fa085223680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:29:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.482+0000 7fa085223680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:29:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.482+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.482+0000 7fa085223680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.482+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.482+0000 7fa085223680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.482+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.488 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.483+0000 7fa085223680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:29:25.488 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.483+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.488 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.483+0000 7fa085223680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.489 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.483+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.489 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.483+0000 7fa085223680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.489 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.483+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.489 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.483+0000 7fa085223680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:25.495 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.489+0000 7fa085223680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:29:25.496 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.489+0000 7fa085223680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:29:25.496 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.489+0000 7fa085223680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:29:25.497 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.489+0000 7fa085223680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:29:25.497 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.489+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.497 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.490+0000 7fa085223680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:25.498 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.490+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.498 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.490+0000 7fa085223680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:29:25.498 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.490+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.499 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.490+0000 7fa085223680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:25.499 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.490+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.499 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.490+0000 7fa085223680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:29:25.500 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.490+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.500 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.500 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 20 zone default found 2023-01-24T17:29:25.501 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 4 Realm: () 2023-01-24T17:29:25.501 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 4 ZoneGroup: default (ea8e6aae-de17-4da7-a2fe-dcef04325757) 2023-01-24T17:29:25.501 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 4 Zone: default (60550c17-6c7c-4ff3-b915-efe39561b124) 2023-01-24T17:29:25.502 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:29:25.502 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 20 zonegroup default 2023-01-24T17:29:25.502 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.502 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.503 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.503 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.503 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.503 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:25.503 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 20 started sync module instance, tier type = 2023-01-24T17:29:25.504 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.491+0000 7fa085223680 20 started zone id=60550c17-6c7c-4ff3-b915-efe39561b124 (name=default) with tier type = 2023-01-24T17:29:25.504 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.501+0000 7fa085223680 20 add_watcher() i=0 2023-01-24T17:29:25.505 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.501+0000 7fa085223680 20 add_watcher() i=1 2023-01-24T17:29:25.505 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.501+0000 7fa085223680 20 add_watcher() i=2 2023-01-24T17:29:25.505 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.501+0000 7fa085223680 20 add_watcher() i=3 2023-01-24T17:29:25.505 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.501+0000 7fa085223680 20 add_watcher() i=4 2023-01-24T17:29:25.506 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.501+0000 7fa085223680 20 add_watcher() i=5 2023-01-24T17:29:25.506 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.501+0000 7fa085223680 20 add_watcher() i=6 2023-01-24T17:29:25.506 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.502+0000 7fa085223680 20 add_watcher() i=7 2023-01-24T17:29:25.506 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.502+0000 7fa085223680 2 all 8 watchers are set, enabling cache 2023-01-24T17:29:25.507 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.503+0000 7fa049fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:29:25.507 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.503+0000 7fa085223680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:29:25.507 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.503+0000 7fa085223680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:29:25.509 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.507+0000 7fa085223680 5 note: GC not initialized 2023-01-24T17:29:25.511 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.507+0000 7f9fc0ff9700 20 reqs_thread_entry: start 2023-01-24T17:29:25.511 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7f9fb67fc700 20 reqs_thread_entry: start 2023-01-24T17:29:25.511 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7fa085223680 20 init_complete bucket index max shards: 11 2023-01-24T17:29:25.511 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7fa085223680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:29:25.512 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7f9fb4ff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:29:55 2023 2023-01-24T17:29:25.512 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-01-24T17:29:25.512 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7f9f9bfff700 20 reqs_thread_entry: start 2023-01-24T17:29:25.512 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7fa085223680 20 get_system_obj_state: rctx=0x7ffed7edbe88 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55f0dffe1940 s->prefetch_data=0 2023-01-24T17:29:25.512 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7fa085223680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-01-24T17:29:25.513 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7fa085223680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-01-24T17:29:25.513 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7fa085223680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:29:25.513 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7fa085223680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:29:25.513 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7fa085223680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:29:25.513 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7fa085223680 20 Read xattr: user.rgw.idtag 2023-01-24T17:29:25.514 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7fa085223680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:29:25.514 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.508+0000 7fa085223680 20 rados->read ofs=0 len=0 2023-01-24T17:29:25.514 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.509+0000 7fa085223680 20 rados_obj.operate() r=0 bl.length=455 2023-01-24T17:29:25.515 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.509+0000 7fa085223680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-01-24T17:29:25.515 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.509+0000 7fa085223680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:29:25.515 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:25.509+0000 7fa085223680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:29:27.546 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.544+0000 7fa085223680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T17:29:27.547 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.544+0000 7fa085223680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:29:27.547 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.544+0000 7fa085223680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-01-24T17:29:27.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.544+0000 7fa075b20700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 94493040087424 notifier 4225 bl.length()=654 2023-01-24T17:29:27.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.544+0000 7fa075b20700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T17:29:27.548 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.544+0000 7fa075b20700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:29:27.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.547+0000 7fa085223680 20 remove_watcher() i=0 2023-01-24T17:29:27.549 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.547+0000 7fa085223680 2 removed watcher, disabling cache 2023-01-24T17:29:27.550 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.548+0000 7fa085223680 20 remove_watcher() i=1 2023-01-24T17:29:27.552 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.549+0000 7fa085223680 20 remove_watcher() i=2 2023-01-24T17:29:27.554 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.552+0000 7fa085223680 20 remove_watcher() i=3 2023-01-24T17:29:27.557 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.555+0000 7fa085223680 20 remove_watcher() i=4 2023-01-24T17:29:27.558 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.556+0000 7fa085223680 20 remove_watcher() i=5 2023-01-24T17:29:27.559 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.557+0000 7fa085223680 20 remove_watcher() i=6 2023-01-24T17:29:27.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.558+0000 7fa085223680 20 remove_watcher() i=7 2023-01-24T17:29:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.559+0000 7fa049fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:29:27.567 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2023-01-24T17:29:27.567 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key AJCMEGZGFSBXIBVXVBMR --secret Y43k0unCDYzV9+02wK0klMuXlW2v9nToVQYw40dby48AczU7Z2ch+Q== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-01-24T17:29:27.618 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:29:27.618 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:29:27.623 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.621+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.625 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.622+0000 7f4c9f19d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.622+0000 7f4c9f19d680 20 realm 2023-01-24T17:29:27.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.622+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.622+0000 7f4c9f19d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.622+0000 7f4c9f19d680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:29:27.627 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.622+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.627 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.622+0000 7f4c9f19d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.627 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.622+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.627 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.623+0000 7f4c9f19d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.627 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.623+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.628 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.623+0000 7f4c9f19d680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:29:27.628 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.623+0000 7f4c9f19d680 20 searching for the correct realm 2023-01-24T17:29:27.636 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.631+0000 7f4c9f19d680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:29:27.636 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.631+0000 7f4c9f19d680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:29:27.636 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.631+0000 7f4c9f19d680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:29:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.631+0000 7f4c9f19d680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:29:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.631+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.631+0000 7f4c9f19d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.631+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.631+0000 7f4c9f19d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.631+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.639 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.632+0000 7f4c9f19d680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:29:27.639 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.632+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.639 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.632+0000 7f4c9f19d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.639 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.632+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.639 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.632+0000 7f4c9f19d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.639 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.632+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.640 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.632+0000 7f4c9f19d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.642 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.639+0000 7f4c9f19d680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:29:27.642 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.639+0000 7f4c9f19d680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:29:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.639+0000 7f4c9f19d680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:29:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.639+0000 7f4c9f19d680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:29:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.639+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.639+0000 7f4c9f19d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.639+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.639+0000 7f4c9f19d680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:29:27.644 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.639+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.644 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.639+0000 7f4c9f19d680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.644 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.639+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.644 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:29:27.645 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.646 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.646 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 20 zone default found 2023-01-24T17:29:27.646 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 4 Realm: () 2023-01-24T17:29:27.646 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 4 ZoneGroup: default (ea8e6aae-de17-4da7-a2fe-dcef04325757) 2023-01-24T17:29:27.646 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 4 Zone: default (60550c17-6c7c-4ff3-b915-efe39561b124) 2023-01-24T17:29:27.647 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:29:27.647 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 20 zonegroup default 2023-01-24T17:29:27.647 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.647 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.647 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.647 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.648 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.640+0000 7f4c9f19d680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.648 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.641+0000 7f4c9f19d680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.648 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.641+0000 7f4c9f19d680 20 started sync module instance, tier type = 2023-01-24T17:29:27.648 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.641+0000 7f4c9f19d680 20 started zone id=60550c17-6c7c-4ff3-b915-efe39561b124 (name=default) with tier type = 2023-01-24T17:29:27.661 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.659+0000 7f4c9f19d680 20 add_watcher() i=0 2023-01-24T17:29:27.662 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.659+0000 7f4c9f19d680 20 add_watcher() i=1 2023-01-24T17:29:27.662 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.659+0000 7f4c9f19d680 20 add_watcher() i=2 2023-01-24T17:29:27.662 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.659+0000 7f4c9f19d680 20 add_watcher() i=3 2023-01-24T17:29:27.663 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.659+0000 7f4c9f19d680 20 add_watcher() i=4 2023-01-24T17:29:27.663 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.659+0000 7f4c9f19d680 20 add_watcher() i=5 2023-01-24T17:29:27.663 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.659+0000 7f4c9f19d680 20 add_watcher() i=6 2023-01-24T17:29:27.664 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.661+0000 7f4c9f19d680 20 add_watcher() i=7 2023-01-24T17:29:27.664 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.661+0000 7f4c9f19d680 2 all 8 watchers are set, enabling cache 2023-01-24T17:29:27.665 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.662+0000 7f4c63fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:29:27.665 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.662+0000 7f4c9f19d680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:29:27.665 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.662+0000 7f4c9f19d680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:29:27.669 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.666+0000 7f4c9f19d680 5 note: GC not initialized 2023-01-24T17:29:27.671 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.667+0000 7f4bd2ffd700 20 reqs_thread_entry: start 2023-01-24T17:29:27.671 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.667+0000 7f4bd0ff9700 20 reqs_thread_entry: start 2023-01-24T17:29:27.672 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.667+0000 7f4c9f19d680 20 init_complete bucket index max shards: 11 2023-01-24T17:29:27.672 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.667+0000 7f4c9f19d680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:29:27.672 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.667+0000 7f4bc6ffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:29:58 2023 2023-01-24T17:29:27.672 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-01-24T17:29:27.672 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4bc67fc700 20 reqs_thread_entry: start 2023-01-24T17:29:27.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 20 get_system_obj_state: rctx=0x7ffc60850398 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55f7baf62940 s->prefetch_data=0 2023-01-24T17:29:27.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-01-24T17:29:27.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2023-01-24T17:29:27.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:29:27.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 20 get_system_obj_state: rctx=0x7ffc60850088 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x55f7baf62940 s->prefetch_data=0 2023-01-24T17:29:27.674 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2023-01-24T17:29:27.674 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2023-01-24T17:29:27.674 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-01-24T17:29:27.674 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 20 get_system_obj_state: rctx=0x7ffc60850088 obj=default.rgw.meta:users.keys:AJCMEGZGFSBXIBVXVBMR state=0x55f7baf62940 s->prefetch_data=0 2023-01-24T17:29:27.674 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 10 cache get: name=default.rgw.meta+users.keys+AJCMEGZGFSBXIBVXVBMR : miss 2023-01-24T17:29:27.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 10 cache put: name=default.rgw.meta+users.keys+AJCMEGZGFSBXIBVXVBMR info.flags=0x0 2023-01-24T17:29:27.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 10 adding default.rgw.meta+users.keys+AJCMEGZGFSBXIBVXVBMR to cache LRU end 2023-01-24T17:29:27.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 20 get_system_obj_state: rctx=0x7ffc6084f918 obj=default.rgw.meta:users.keys:AJCMEGZGFSBXIBVXVBMR state=0x55f7baf62940 s->prefetch_data=0 2023-01-24T17:29:27.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 10 cache get: name=default.rgw.meta+users.keys+AJCMEGZGFSBXIBVXVBMR : hit (negative entry) 2023-01-24T17:29:27.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 20 get_system_obj_state: rctx=0x7ffc6084ffc8 obj=default.rgw.meta:users.keys:AJCMEGZGFSBXIBVXVBMR state=0x55f7baf62940 s->prefetch_data=0 2023-01-24T17:29:27.676 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.668+0000 7f4c9f19d680 10 cache get: name=default.rgw.meta+users.keys+AJCMEGZGFSBXIBVXVBMR : hit (negative entry) 2023-01-24T17:29:27.678 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.670+0000 7f4c9f19d680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T17:29:27.678 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.670+0000 7f4c9f19d680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:29:27.678 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.670+0000 7f4c9f19d680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-01-24T17:29:27.679 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.670+0000 7f4c83fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116993 cookie 94522483585280 notifier 4231 bl.length()=640 2023-01-24T17:29:27.679 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.670+0000 7f4c83fff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T17:29:27.679 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.670+0000 7f4c83fff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:29:27.680 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.671+0000 7f4c9f19d680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-01-24T17:29:27.680 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.671+0000 7f4c9f19d680 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-01-24T17:29:27.680 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.671+0000 7f4c9f19d680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-01-24T17:29:27.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.672+0000 7f4c837fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 94522483585280 notifier 4231 bl.length()=194 2023-01-24T17:29:27.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.672+0000 7f4c837fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-01-24T17:29:27.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.672+0000 7f4c837fe700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-01-24T17:29:27.682 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.673+0000 7f4c9f19d680 10 cache put: name=default.rgw.meta+users.keys+AJCMEGZGFSBXIBVXVBMR info.flags=0x7 2023-01-24T17:29:27.682 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.673+0000 7f4c9f19d680 10 moving default.rgw.meta+users.keys+AJCMEGZGFSBXIBVXVBMR to cache LRU end 2023-01-24T17:29:27.682 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.673+0000 7f4c9f19d680 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.keys:AJCMEGZGFSBXIBVXVBMR, ofs0, ns] 2023-01-24T17:29:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.673+0000 7f4c83fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 94522483582480 notifier 4231 bl.length()=186 2023-01-24T17:29:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.673+0000 7f4c83fff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+AJCMEGZGFSBXIBVXVBMR info.flags=0x7 2023-01-24T17:29:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.673+0000 7f4c83fff700 10 rgw watcher librados: moving default.rgw.meta+users.keys+AJCMEGZGFSBXIBVXVBMR to cache LRU end 2023-01-24T17:29:27.684 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-01-24T17:29:27.684 INFO:teuthology.orchestra.run.smithi150.stdout: "user_id": "bar.client.0", 2023-01-24T17:29:27.684 INFO:teuthology.orchestra.run.smithi150.stdout: "display_name": "Mr. bar.client.0", 2023-01-24T17:29:27.684 INFO:teuthology.orchestra.run.smithi150.stdout: "email": "bar.client.0+test@test.test", 2023-01-24T17:29:27.684 INFO:teuthology.orchestra.run.smithi150.stdout: "suspended": 0, 2023-01-24T17:29:27.685 INFO:teuthology.orchestra.run.smithi150.stdout: "max_buckets": 1000, 2023-01-24T17:29:27.685 INFO:teuthology.orchestra.run.smithi150.stdout: "subusers": [], 2023-01-24T17:29:27.685 INFO:teuthology.orchestra.run.smithi150.stdout: "keys": [ 2023-01-24T17:29:27.685 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-01-24T17:29:27.685 INFO:teuthology.orchestra.run.smithi150.stdout: "user": "bar.client.0", 2023-01-24T17:29:27.686 INFO:teuthology.orchestra.run.smithi150.stdout: "access_key": "AJCMEGZGFSBXIBVXVBMR", 2023-01-24T17:29:27.686 INFO:teuthology.orchestra.run.smithi150.stdout: "secret_key": "Y43k0unCDYzV9+02wK0klMuXlW2v9nToVQYw40dby48AczU7Z2ch+Q==" 2023-01-24T17:29:27.686 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-01-24T17:29:27.686 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2023-01-24T17:29:27.686 INFO:teuthology.orchestra.run.smithi150.stdout: "swift_keys": [], 2023-01-24T17:29:27.687 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": [ 2023-01-24T17:29:27.687 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-01-24T17:29:27.687 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "user-policy", 2023-01-24T17:29:27.687 INFO:teuthology.orchestra.run.smithi150.stdout: "perm": "*" 2023-01-24T17:29:27.687 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-01-24T17:29:27.688 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2023-01-24T17:29:27.688 INFO:teuthology.orchestra.run.smithi150.stdout: "op_mask": "read, write, delete", 2023-01-24T17:29:27.688 INFO:teuthology.orchestra.run.smithi150.stdout: "default_placement": "", 2023-01-24T17:29:27.688 INFO:teuthology.orchestra.run.smithi150.stdout: "default_storage_class": "", 2023-01-24T17:29:27.688 INFO:teuthology.orchestra.run.smithi150.stdout: "placement_tags": [], 2023-01-24T17:29:27.689 INFO:teuthology.orchestra.run.smithi150.stdout: "bucket_quota": { 2023-01-24T17:29:27.689 INFO:teuthology.orchestra.run.smithi150.stdout: "enabled": false, 2023-01-24T17:29:27.689 INFO:teuthology.orchestra.run.smithi150.stdout: "check_on_raw": false, 2023-01-24T17:29:27.689 INFO:teuthology.orchestra.run.smithi150.stdout: "max_size": -1, 2023-01-24T17:29:27.689 INFO:teuthology.orchestra.run.smithi150.stdout: "max_size_kb": 0, 2023-01-24T17:29:27.689 INFO:teuthology.orchestra.run.smithi150.stdout: "max_objects": -1 2023-01-24T17:29:27.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-01-24T17:29:27.690 INFO:teuthology.orchestra.run.smithi150.stdout: "user_quota": { 2023-01-24T17:29:27.690 INFO:teuthology.orchestra.run.smithi150.stdout: "enabled": false, 2023-01-24T17:29:27.690 INFO:teuthology.orchestra.run.smithi150.stdout: "check_on_raw": false, 2023-01-24T17:29:27.690 INFO:teuthology.orchestra.run.smithi150.stdout: "max_size": -1, 2023-01-24T17:29:27.691 INFO:teuthology.orchestra.run.smithi150.stdout: "max_size_kb": 0, 2023-01-24T17:29:27.691 INFO:teuthology.orchestra.run.smithi150.stdout: "max_objects": -1 2023-01-24T17:29:27.691 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-01-24T17:29:27.691 INFO:teuthology.orchestra.run.smithi150.stdout: "temp_url_keys": [], 2023-01-24T17:29:27.691 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "rgw", 2023-01-24T17:29:27.692 INFO:teuthology.orchestra.run.smithi150.stdout: "mfa_ids": [] 2023-01-24T17:29:27.692 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-01-24T17:29:27.692 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:29:27.694 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.675+0000 7f4c9f19d680 20 remove_watcher() i=0 2023-01-24T17:29:27.694 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.675+0000 7f4c9f19d680 2 removed watcher, disabling cache 2023-01-24T17:29:27.694 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.675+0000 7f4c9f19d680 20 remove_watcher() i=1 2023-01-24T17:29:27.694 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.676+0000 7f4c9f19d680 20 remove_watcher() i=2 2023-01-24T17:29:27.695 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.677+0000 7f4c9f19d680 20 remove_watcher() i=3 2023-01-24T17:29:27.695 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.678+0000 7f4c9f19d680 20 remove_watcher() i=4 2023-01-24T17:29:27.695 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.679+0000 7f4c9f19d680 20 remove_watcher() i=5 2023-01-24T17:29:27.695 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.680+0000 7f4c9f19d680 20 remove_watcher() i=6 2023-01-24T17:29:27.695 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.681+0000 7f4c9f19d680 20 remove_watcher() i=7 2023-01-24T17:29:27.696 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.681+0000 7f4c63fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:29:27.696 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 3263152267 --totp-seed MWQTKLVWW2DUXJY3CHQKD2XLAOSDP43WDLNHJTTCEAECNRNG2FX67Q2S73Q4BAK2 --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-01-24T17:29:27.744 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:29:27.744 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:29:27.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.747+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.747+0000 7f9515bc1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.747+0000 7f9515bc1680 20 realm 2023-01-24T17:29:27.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.747+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.748+0000 7f9515bc1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.748+0000 7f9515bc1680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:29:27.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.748+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.748+0000 7f9515bc1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.748+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.748+0000 7f9515bc1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.748+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.749+0000 7f9515bc1680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:29:27.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.749+0000 7f9515bc1680 20 searching for the correct realm 2023-01-24T17:29:27.759 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.755+0000 7f9515bc1680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:29:27.760 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.755+0000 7f9515bc1680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:29:27.760 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.755+0000 7f9515bc1680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:29:27.760 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.755+0000 7f9515bc1680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:29:27.761 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.755+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.761 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.756+0000 7f9515bc1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.761 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.756+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.761 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.756+0000 7f9515bc1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.761 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.756+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.761 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.756+0000 7f9515bc1680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:29:27.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.756+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.756+0000 7f9515bc1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.756+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.756+0000 7f9515bc1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.756+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.763 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.757+0000 7f9515bc1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.762+0000 7f9515bc1680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:29:27.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.762+0000 7f9515bc1680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:29:27.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.762+0000 7f9515bc1680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:29:27.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.762+0000 7f9515bc1680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:29:27.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.763+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.763+0000 7f9515bc1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.763+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.769 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.763+0000 7f9515bc1680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:29:27.769 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.763+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.769 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.763+0000 7f9515bc1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.769 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.763+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.769 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.763+0000 7f9515bc1680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:29:27.770 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.763+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.770 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.770 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 20 zone default found 2023-01-24T17:29:27.770 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 4 Realm: () 2023-01-24T17:29:27.771 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 4 ZoneGroup: default (ea8e6aae-de17-4da7-a2fe-dcef04325757) 2023-01-24T17:29:27.771 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 4 Zone: default (60550c17-6c7c-4ff3-b915-efe39561b124) 2023-01-24T17:29:27.771 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:29:27.771 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 20 zonegroup default 2023-01-24T17:29:27.771 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.772 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.772 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.772 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.772 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.773 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.773 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 20 started sync module instance, tier type = 2023-01-24T17:29:27.773 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.764+0000 7f9515bc1680 20 started zone id=60550c17-6c7c-4ff3-b915-efe39561b124 (name=default) with tier type = 2023-01-24T17:29:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.780+0000 7f9515bc1680 20 add_watcher() i=0 2023-01-24T17:29:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.780+0000 7f9515bc1680 20 add_watcher() i=1 2023-01-24T17:29:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.780+0000 7f9515bc1680 20 add_watcher() i=2 2023-01-24T17:29:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.780+0000 7f9515bc1680 20 add_watcher() i=3 2023-01-24T17:29:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.780+0000 7f9515bc1680 20 add_watcher() i=4 2023-01-24T17:29:27.784 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.780+0000 7f9515bc1680 20 add_watcher() i=5 2023-01-24T17:29:27.784 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.782+0000 7f9515bc1680 20 add_watcher() i=6 2023-01-24T17:29:27.785 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.782+0000 7f9515bc1680 20 add_watcher() i=7 2023-01-24T17:29:27.785 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.782+0000 7f9515bc1680 2 all 8 watchers are set, enabling cache 2023-01-24T17:29:27.786 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.783+0000 7f94d9fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:29:27.786 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.783+0000 7f9515bc1680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:29:27.786 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.783+0000 7f9515bc1680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:29:27.789 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.787+0000 7f9515bc1680 5 note: GC not initialized 2023-01-24T17:29:27.792 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9450ff9700 20 reqs_thread_entry: start 2023-01-24T17:29:27.792 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f94467fc700 20 reqs_thread_entry: start 2023-01-24T17:29:27.793 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9515bc1680 20 init_complete bucket index max shards: 11 2023-01-24T17:29:27.793 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9515bc1680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:29:27.793 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9444ff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:29:57 2023 2023-01-24T17:29:27.794 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-01-24T17:29:27.794 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f942bfff700 20 reqs_thread_entry: start 2023-01-24T17:29:27.794 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9515bc1680 20 get_system_obj_state: rctx=0x7ffda5d39db8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x560f8a247940 s->prefetch_data=0 2023-01-24T17:29:27.794 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9515bc1680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-01-24T17:29:27.794 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9515bc1680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-01-24T17:29:27.795 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9515bc1680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:29:27.795 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9515bc1680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:29:27.795 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9515bc1680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:29:27.795 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9515bc1680 20 Read xattr: user.rgw.idtag 2023-01-24T17:29:27.795 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9515bc1680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:29:27.796 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.788+0000 7f9515bc1680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.796 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.789+0000 7f9515bc1680 20 rados_obj.operate() r=0 bl.length=455 2023-01-24T17:29:27.796 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.789+0000 7f9515bc1680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-01-24T17:29:27.796 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.789+0000 7f9515bc1680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:29:27.796 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.789+0000 7f9515bc1680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:29:27.797 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.793+0000 7f9515bc1680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T17:29:27.798 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.793+0000 7f9515bc1680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:29:27.798 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.793+0000 7f9515bc1680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-01-24T17:29:27.798 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.793+0000 7f95027fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94624743746832 notifier 4237 bl.length()=654 2023-01-24T17:29:27.798 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.793+0000 7f95027fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T17:29:27.799 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.793+0000 7f95027fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:29:27.799 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.794+0000 7f9515bc1680 20 remove_watcher() i=0 2023-01-24T17:29:27.799 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.794+0000 7f9515bc1680 2 removed watcher, disabling cache 2023-01-24T17:29:27.799 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.795+0000 7f9515bc1680 20 remove_watcher() i=1 2023-01-24T17:29:27.800 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.798+0000 7f9515bc1680 20 remove_watcher() i=2 2023-01-24T17:29:27.803 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.801+0000 7f9515bc1680 20 remove_watcher() i=3 2023-01-24T17:29:27.804 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.802+0000 7f9515bc1680 20 remove_watcher() i=4 2023-01-24T17:29:27.806 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.804+0000 7f9515bc1680 20 remove_watcher() i=5 2023-01-24T17:29:27.807 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.804+0000 7f9515bc1680 20 remove_watcher() i=6 2023-01-24T17:29:27.808 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.805+0000 7f9515bc1680 20 remove_watcher() i=7 2023-01-24T17:29:27.808 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.806+0000 7f94d9fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:29:27.813 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2023-01-24T17:29:27.813 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key DSPNUPXTHAQPETOJGKQM --secret quQCelgn5O14ewXIm3kQMpNPau6rpw5Ke1u7Ad+T9gUWZVisICNXIg== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2023-01-24T17:29:27.861 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:29:27.861 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:29:27.866 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.863+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.867 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.864+0000 7f12c1b79680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.868 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.864+0000 7f12c1b79680 20 realm 2023-01-24T17:29:27.868 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.864+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.868 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.865+0000 7f12c1b79680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.868 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.865+0000 7f12c1b79680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:29:27.869 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.865+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.869 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.865+0000 7f12c1b79680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.869 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.865+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.870 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.865+0000 7f12c1b79680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.870 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.865+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.870 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.866+0000 7f12c1b79680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:29:27.870 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.866+0000 7f12c1b79680 20 searching for the correct realm 2023-01-24T17:29:27.876 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:29:27.876 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:29:27.876 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:29:27.877 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:29:27.877 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.877 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.877 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.878 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.878 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.878 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:29:27.878 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.878 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.879 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.873+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.879 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.874+0000 7f12c1b79680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.879 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.874+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.880 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.874+0000 7f12c1b79680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.883 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.880+0000 7f12c1b79680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:29:27.883 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.880+0000 7f12c1b79680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:29:27.883 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.880+0000 7f12c1b79680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:29:27.884 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.880+0000 7f12c1b79680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:29:27.884 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.880+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.884 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.880+0000 7f12c1b79680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.884 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.880+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.885 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.880+0000 7f12c1b79680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:29:27.885 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.880+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.885 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:27.885 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.886 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:29:27.887 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.887 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.887 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 20 zone default found 2023-01-24T17:29:27.887 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 4 Realm: () 2023-01-24T17:29:27.888 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 4 ZoneGroup: default (ea8e6aae-de17-4da7-a2fe-dcef04325757) 2023-01-24T17:29:27.888 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 4 Zone: default (60550c17-6c7c-4ff3-b915-efe39561b124) 2023-01-24T17:29:27.888 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:29:27.888 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 20 zonegroup default 2023-01-24T17:29:27.888 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.889 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.889 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.889 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.889 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.881+0000 7f12c1b79680 20 rados->read ofs=0 len=0 2023-01-24T17:29:27.889 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.882+0000 7f12c1b79680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:27.890 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.882+0000 7f12c1b79680 20 started sync module instance, tier type = 2023-01-24T17:29:27.890 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.882+0000 7f12c1b79680 20 started zone id=60550c17-6c7c-4ff3-b915-efe39561b124 (name=default) with tier type = 2023-01-24T17:29:27.909 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.906+0000 7f12c1b79680 20 add_watcher() i=0 2023-01-24T17:29:27.909 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.906+0000 7f12c1b79680 20 add_watcher() i=1 2023-01-24T17:29:27.910 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.906+0000 7f12c1b79680 20 add_watcher() i=2 2023-01-24T17:29:27.910 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.906+0000 7f12c1b79680 20 add_watcher() i=3 2023-01-24T17:29:27.910 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.906+0000 7f12c1b79680 20 add_watcher() i=4 2023-01-24T17:29:27.910 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.906+0000 7f12c1b79680 20 add_watcher() i=5 2023-01-24T17:29:27.910 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.906+0000 7f12c1b79680 20 add_watcher() i=6 2023-01-24T17:29:27.911 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.909+0000 7f12c1b79680 20 add_watcher() i=7 2023-01-24T17:29:27.911 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.909+0000 7f12c1b79680 2 all 8 watchers are set, enabling cache 2023-01-24T17:29:27.913 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.910+0000 7f1285fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:29:27.913 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.910+0000 7f12c1b79680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:29:27.913 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.910+0000 7f12c1b79680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:29:27.916 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.914+0000 7f12c1b79680 5 note: GC not initialized 2023-01-24T17:29:27.919 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.915+0000 7f11fcff9700 20 reqs_thread_entry: start 2023-01-24T17:29:27.919 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.915+0000 7f11f27fc700 20 reqs_thread_entry: start 2023-01-24T17:29:27.919 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.915+0000 7f12c1b79680 20 init_complete bucket index max shards: 11 2023-01-24T17:29:27.920 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.915+0000 7f12c1b79680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:29:27.920 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.915+0000 7f11f0ff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:29:58 2023 2023-01-24T17:29:27.920 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-01-24T17:29:27.920 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.915+0000 7f11d7fff700 20 reqs_thread_entry: start 2023-01-24T17:29:27.920 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.915+0000 7f12c1b79680 20 get_system_obj_state: rctx=0x7ffdb08ebaa8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x56381243f940 s->prefetch_data=0 2023-01-24T17:29:27.921 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.915+0000 7f12c1b79680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-01-24T17:29:27.921 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2023-01-24T17:29:27.921 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:29:27.921 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 20 get_system_obj_state: rctx=0x7ffdb08eb798 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x56381243f940 s->prefetch_data=0 2023-01-24T17:29:27.921 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2023-01-24T17:29:27.922 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2023-01-24T17:29:27.922 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-01-24T17:29:27.922 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 20 get_system_obj_state: rctx=0x7ffdb08eb798 obj=default.rgw.meta:users.keys:DSPNUPXTHAQPETOJGKQM state=0x56381243f940 s->prefetch_data=0 2023-01-24T17:29:27.922 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 10 cache get: name=default.rgw.meta+users.keys+DSPNUPXTHAQPETOJGKQM : miss 2023-01-24T17:29:27.922 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 10 cache put: name=default.rgw.meta+users.keys+DSPNUPXTHAQPETOJGKQM info.flags=0x0 2023-01-24T17:29:27.922 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 10 adding default.rgw.meta+users.keys+DSPNUPXTHAQPETOJGKQM to cache LRU end 2023-01-24T17:29:27.923 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 20 get_system_obj_state: rctx=0x7ffdb08eb028 obj=default.rgw.meta:users.keys:DSPNUPXTHAQPETOJGKQM state=0x56381243f940 s->prefetch_data=0 2023-01-24T17:29:27.923 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 10 cache get: name=default.rgw.meta+users.keys+DSPNUPXTHAQPETOJGKQM : hit (negative entry) 2023-01-24T17:29:27.923 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 20 get_system_obj_state: rctx=0x7ffdb08eb6d8 obj=default.rgw.meta:users.keys:DSPNUPXTHAQPETOJGKQM state=0x56381243f940 s->prefetch_data=0 2023-01-24T17:29:27.923 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.916+0000 7f12c1b79680 10 cache get: name=default.rgw.meta+users.keys+DSPNUPXTHAQPETOJGKQM : hit (negative entry) 2023-01-24T17:29:27.924 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.917+0000 7f12c1b79680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T17:29:27.925 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.917+0000 7f12c1b79680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:29:27.925 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.917+0000 7f12c1b79680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-01-24T17:29:27.925 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.918+0000 7f12ae7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 94798826222656 notifier 4243 bl.length()=714 2023-01-24T17:29:27.925 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.918+0000 7f12ae7fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T17:29:27.925 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.918+0000 7f12ae7fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:29:27.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.920+0000 7f12c1b79680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-01-24T17:29:27.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.920+0000 7f12c1b79680 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-01-24T17:29:27.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.920+0000 7f12c1b79680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-01-24T17:29:27.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.920+0000 7f12adffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94798826230160 notifier 4243 bl.length()=224 2023-01-24T17:29:27.926 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.920+0000 7f12adffb700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-01-24T17:29:27.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.920+0000 7f12adffb700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-01-24T17:29:27.927 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.923+0000 7f12c1b79680 10 cache put: name=default.rgw.meta+users.keys+DSPNUPXTHAQPETOJGKQM info.flags=0x7 2023-01-24T17:29:27.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.923+0000 7f12c1b79680 10 moving default.rgw.meta+users.keys+DSPNUPXTHAQPETOJGKQM to cache LRU end 2023-01-24T17:29:27.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.923+0000 7f12c1b79680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:DSPNUPXTHAQPETOJGKQM, ofs0, ns] 2023-01-24T17:29:27.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.923+0000 7f12adffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116997 cookie 94798826209248 notifier 4243 bl.length()=201 2023-01-24T17:29:27.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.923+0000 7f12adffb700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+DSPNUPXTHAQPETOJGKQM info.flags=0x7 2023-01-24T17:29:27.928 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.923+0000 7f12adffb700 10 rgw watcher librados: moving default.rgw.meta+users.keys+DSPNUPXTHAQPETOJGKQM to cache LRU end 2023-01-24T17:29:27.929 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-01-24T17:29:27.929 INFO:teuthology.orchestra.run.smithi150.stdout: "user_id": "testx$tenanteduser.client.0", 2023-01-24T17:29:27.929 INFO:teuthology.orchestra.run.smithi150.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2023-01-24T17:29:27.929 INFO:teuthology.orchestra.run.smithi150.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2023-01-24T17:29:27.930 INFO:teuthology.orchestra.run.smithi150.stdout: "suspended": 0, 2023-01-24T17:29:27.930 INFO:teuthology.orchestra.run.smithi150.stdout: "max_buckets": 1000, 2023-01-24T17:29:27.930 INFO:teuthology.orchestra.run.smithi150.stdout: "subusers": [], 2023-01-24T17:29:27.930 INFO:teuthology.orchestra.run.smithi150.stdout: "keys": [ 2023-01-24T17:29:27.930 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-01-24T17:29:27.931 INFO:teuthology.orchestra.run.smithi150.stdout: "user": "testx$tenanteduser.client.0", 2023-01-24T17:29:27.931 INFO:teuthology.orchestra.run.smithi150.stdout: "access_key": "DSPNUPXTHAQPETOJGKQM", 2023-01-24T17:29:27.931 INFO:teuthology.orchestra.run.smithi150.stdout: "secret_key": "quQCelgn5O14ewXIm3kQMpNPau6rpw5Ke1u7Ad+T9gUWZVisICNXIg==" 2023-01-24T17:29:27.931 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-01-24T17:29:27.931 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2023-01-24T17:29:27.932 INFO:teuthology.orchestra.run.smithi150.stdout: "swift_keys": [], 2023-01-24T17:29:27.932 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": [ 2023-01-24T17:29:27.932 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-01-24T17:29:27.932 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "user-policy", 2023-01-24T17:29:27.932 INFO:teuthology.orchestra.run.smithi150.stdout: "perm": "*" 2023-01-24T17:29:27.933 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-01-24T17:29:27.933 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2023-01-24T17:29:27.933 INFO:teuthology.orchestra.run.smithi150.stdout: "op_mask": "read, write, delete", 2023-01-24T17:29:27.933 INFO:teuthology.orchestra.run.smithi150.stdout: "default_placement": "", 2023-01-24T17:29:27.933 INFO:teuthology.orchestra.run.smithi150.stdout: "default_storage_class": "", 2023-01-24T17:29:27.934 INFO:teuthology.orchestra.run.smithi150.stdout: "placement_tags": [], 2023-01-24T17:29:27.934 INFO:teuthology.orchestra.run.smithi150.stdout: "bucket_quota": { 2023-01-24T17:29:27.934 INFO:teuthology.orchestra.run.smithi150.stdout: "enabled": false, 2023-01-24T17:29:27.934 INFO:teuthology.orchestra.run.smithi150.stdout: "check_on_raw": false, 2023-01-24T17:29:27.934 INFO:teuthology.orchestra.run.smithi150.stdout: "max_size": -1, 2023-01-24T17:29:27.935 INFO:teuthology.orchestra.run.smithi150.stdout: "max_size_kb": 0, 2023-01-24T17:29:27.935 INFO:teuthology.orchestra.run.smithi150.stdout: "max_objects": -1 2023-01-24T17:29:27.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-01-24T17:29:27.935 INFO:teuthology.orchestra.run.smithi150.stdout: "user_quota": { 2023-01-24T17:29:27.935 INFO:teuthology.orchestra.run.smithi150.stdout: "enabled": false, 2023-01-24T17:29:27.935 INFO:teuthology.orchestra.run.smithi150.stdout: "check_on_raw": false, 2023-01-24T17:29:27.936 INFO:teuthology.orchestra.run.smithi150.stdout: "max_size": -1, 2023-01-24T17:29:27.936 INFO:teuthology.orchestra.run.smithi150.stdout: "max_size_kb": 0, 2023-01-24T17:29:27.936 INFO:teuthology.orchestra.run.smithi150.stdout: "max_objects": -1 2023-01-24T17:29:27.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-01-24T17:29:27.937 INFO:teuthology.orchestra.run.smithi150.stdout: "temp_url_keys": [], 2023-01-24T17:29:27.937 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "rgw", 2023-01-24T17:29:27.937 INFO:teuthology.orchestra.run.smithi150.stdout: "mfa_ids": [] 2023-01-24T17:29:27.937 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-01-24T17:29:27.937 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:29:27.938 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.925+0000 7f12c1b79680 20 remove_watcher() i=0 2023-01-24T17:29:27.939 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.925+0000 7f12c1b79680 2 removed watcher, disabling cache 2023-01-24T17:29:27.939 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.926+0000 7f12c1b79680 20 remove_watcher() i=1 2023-01-24T17:29:27.939 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.927+0000 7f12c1b79680 20 remove_watcher() i=2 2023-01-24T17:29:27.939 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.930+0000 7f12c1b79680 20 remove_watcher() i=3 2023-01-24T17:29:27.939 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.933+0000 7f12c1b79680 20 remove_watcher() i=4 2023-01-24T17:29:27.940 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.934+0000 7f12c1b79680 20 remove_watcher() i=5 2023-01-24T17:29:27.940 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.935+0000 7f12c1b79680 20 remove_watcher() i=6 2023-01-24T17:29:27.940 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.938+0000 7f12c1b79680 20 remove_watcher() i=7 2023-01-24T17:29:27.941 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.939+0000 7f1285fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:29:27.947 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 1944099940 --totp-seed ESG5V5GAHBLNIAFPPO22XXLN4TJYFUJW2JUDNNZ3PKG2KOHQ3EGFF7DYQDQQ6ML4 --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-01-24T17:29:27.994 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:29:27.994 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:29:27.999 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.996+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.001 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.997+0000 7fe78edf9680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:28.001 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.997+0000 7fe78edf9680 20 realm 2023-01-24T17:29:28.001 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.997+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.001 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.998+0000 7fe78edf9680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:28.001 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.998+0000 7fe78edf9680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:29:28.002 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.998+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.002 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.998+0000 7fe78edf9680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:28.002 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.998+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.002 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.998+0000 7fe78edf9680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:28.002 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.998+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.003 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.999+0000 7fe78edf9680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:29:28.003 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:27.999+0000 7fe78edf9680 20 searching for the correct realm 2023-01-24T17:29:28.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.005+0000 7fe78edf9680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:29:28.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.005+0000 7fe78edf9680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:29:28.010 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.005+0000 7fe78edf9680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:29:28.010 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.005+0000 7fe78edf9680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:29:28.010 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.005+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.010 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.006+0000 7fe78edf9680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:28.010 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.006+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.011 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.006+0000 7fe78edf9680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:28.011 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.006+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.011 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.006+0000 7fe78edf9680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:29:28.011 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.006+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.011 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.006+0000 7fe78edf9680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:28.012 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.006+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.012 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.006+0000 7fe78edf9680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:28.012 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.006+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.012 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.007+0000 7fe78edf9680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:28.016 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.013+0000 7fe78edf9680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:29:28.016 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.013+0000 7fe78edf9680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:29:28.016 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.013+0000 7fe78edf9680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:29:28.016 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.013+0000 7fe78edf9680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:29:28.017 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.013+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.017 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.013+0000 7fe78edf9680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:28.017 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.013+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.017 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.013+0000 7fe78edf9680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:29:28.017 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.013+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.018 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.013+0000 7fe78edf9680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:29:28.018 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.013+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.019 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:29:28.019 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.020 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:28.020 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 zone default found 2023-01-24T17:29:28.020 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 4 Realm: () 2023-01-24T17:29:28.020 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 4 ZoneGroup: default (ea8e6aae-de17-4da7-a2fe-dcef04325757) 2023-01-24T17:29:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 4 Zone: default (60550c17-6c7c-4ff3-b915-efe39561b124) 2023-01-24T17:29:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:29:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 zonegroup default 2023-01-24T17:29:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.022 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:28.022 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.022 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:29:28.022 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 started sync module instance, tier type = 2023-01-24T17:29:28.022 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.014+0000 7fe78edf9680 20 started zone id=60550c17-6c7c-4ff3-b915-efe39561b124 (name=default) with tier type = 2023-01-24T17:29:28.032 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.030+0000 7fe78edf9680 20 add_watcher() i=0 2023-01-24T17:29:28.033 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.030+0000 7fe78edf9680 20 add_watcher() i=1 2023-01-24T17:29:28.033 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.030+0000 7fe78edf9680 20 add_watcher() i=2 2023-01-24T17:29:28.033 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.030+0000 7fe78edf9680 20 add_watcher() i=3 2023-01-24T17:29:28.033 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.030+0000 7fe78edf9680 20 add_watcher() i=4 2023-01-24T17:29:28.033 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.030+0000 7fe78edf9680 20 add_watcher() i=5 2023-01-24T17:29:28.034 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.030+0000 7fe78edf9680 20 add_watcher() i=6 2023-01-24T17:29:28.035 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.032+0000 7fe78edf9680 20 add_watcher() i=7 2023-01-24T17:29:28.035 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.032+0000 7fe78edf9680 2 all 8 watchers are set, enabling cache 2023-01-24T17:29:28.036 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.033+0000 7fe7537de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:29:28.036 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.034+0000 7fe78edf9680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:29:28.037 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.034+0000 7fe78edf9680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:29:28.040 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.037+0000 7fe78edf9680 5 note: GC not initialized 2023-01-24T17:29:28.042 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.038+0000 7fe6c27fc700 20 reqs_thread_entry: start 2023-01-24T17:29:28.042 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.038+0000 7fe6b3fff700 20 reqs_thread_entry: start 2023-01-24T17:29:28.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.038+0000 7fe78edf9680 20 init_complete bucket index max shards: 11 2023-01-24T17:29:28.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:29:28.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe6b27fc700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:29:58 2023 2023-01-24T17:29:28.043 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-01-24T17:29:28.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe6b1ffb700 20 reqs_thread_entry: start 2023-01-24T17:29:28.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 20 get_system_obj_state: rctx=0x7ffc0a11d988 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55b52c6b3940 s->prefetch_data=0 2023-01-24T17:29:28.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-01-24T17:29:28.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-01-24T17:29:28.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:29:28.045 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:29:28.045 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:29:28.045 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 20 Read xattr: user.rgw.idtag 2023-01-24T17:29:28.045 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:29:28.045 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 20 rados->read ofs=0 len=0 2023-01-24T17:29:28.045 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 20 rados_obj.operate() r=0 bl.length=514 2023-01-24T17:29:28.046 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-01-24T17:29:28.046 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:29:28.046 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.039+0000 7fe78edf9680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:29:28.047 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.042+0000 7fe78edf9680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T17:29:28.047 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.042+0000 7fe78edf9680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:29:28.048 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.042+0000 7fe78edf9680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-01-24T17:29:28.048 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.042+0000 7fe77b7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116997 cookie 94236624287680 notifier 4249 bl.length()=728 2023-01-24T17:29:28.048 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.042+0000 7fe77b7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T17:29:28.048 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.042+0000 7fe77b7fe700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:29:28.048 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.043+0000 7fe78edf9680 20 remove_watcher() i=0 2023-01-24T17:29:28.049 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.043+0000 7fe78edf9680 2 removed watcher, disabling cache 2023-01-24T17:29:28.049 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.044+0000 7fe78edf9680 20 remove_watcher() i=1 2023-01-24T17:29:28.049 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.045+0000 7fe78edf9680 20 remove_watcher() i=2 2023-01-24T17:29:28.050 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.046+0000 7fe78edf9680 20 remove_watcher() i=3 2023-01-24T17:29:28.050 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.047+0000 7fe78edf9680 20 remove_watcher() i=4 2023-01-24T17:29:28.050 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.048+0000 7fe78edf9680 20 remove_watcher() i=5 2023-01-24T17:29:28.051 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.048+0000 7fe78edf9680 20 remove_watcher() i=6 2023-01-24T17:29:28.052 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.049+0000 7fe78edf9680 20 remove_watcher() i=7 2023-01-24T17:29:28.052 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:29:28.050+0000 7fe7537de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:29:28.057 INFO:tasks.s3tests:Configuring s3-tests... 2023-01-24T17:29:28.057 DEBUG:teuthology.orchestra.run.smithi150:> cd /home/ubuntu/cephtest/s3-tests && ./bootstrap 2023-01-24T17:29:34.851 INFO:teuthology.orchestra.run.smithi150.stderr:./bootstrap: Missing required RPM packages: libevent-devel libffi-devel libxml2-devel libxslt-devel zlib-devel. 2023-01-24T17:29:35.208 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-01-24T17:29:36.003 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:05:02 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:29:37.775 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-01-24T17:29:37.776 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:29:37.777 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-01-24T17:29:37.777 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:29:37.777 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-01-24T17:29:37.777 INFO:teuthology.orchestra.run.smithi150.stdout: libevent-devel x86_64 2.1.8-5.el8 rhel-8-for-x86_64-appstream-rpms 104 k 2023-01-24T17:29:37.777 INFO:teuthology.orchestra.run.smithi150.stdout: libffi-devel x86_64 3.1-22.el8 rhel-8-for-x86_64-baseos-rpms 29 k 2023-01-24T17:29:37.778 INFO:teuthology.orchestra.run.smithi150.stdout: libxml2-devel x86_64 2.9.7-9.el8_4.2 rhel-8-for-x86_64-appstream-rpms 1.0 M 2023-01-24T17:29:37.778 INFO:teuthology.orchestra.run.smithi150.stdout: libxslt-devel x86_64 1.1.32-6.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-01-24T17:29:37.778 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-01-24T17:29:37.779 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-01-24T17:29:37.779 INFO:teuthology.orchestra.run.smithi150.stdout: cmake-filesystem x86_64 3.18.2-11.el8_4 rhel-8-for-x86_64-appstream-rpms 44 k 2023-01-24T17:29:37.779 INFO:teuthology.orchestra.run.smithi150.stdout: libgcrypt-devel x86_64 1.8.5-4.el8 rhel-8-for-x86_64-baseos-rpms 149 k 2023-01-24T17:29:37.779 INFO:teuthology.orchestra.run.smithi150.stdout: libgpg-error-devel 2023-01-24T17:29:37.780 INFO:teuthology.orchestra.run.smithi150.stdout: x86_64 1.31-1.el8 rhel-8-for-x86_64-baseos-rpms 64 k 2023-01-24T17:29:37.780 INFO:teuthology.orchestra.run.smithi150.stdout: xz-devel x86_64 5.2.4-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2023-01-24T17:29:37.780 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:29:37.780 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-01-24T17:29:37.780 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-01-24T17:29:37.781 INFO:teuthology.orchestra.run.smithi150.stdout:Install 9 Packages 2023-01-24T17:29:37.781 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:29:37.782 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 1.9 M 2023-01-24T17:29:37.782 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 12 M 2023-01-24T17:29:37.782 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-01-24T17:29:37.833 INFO:teuthology.orchestra.run.smithi150.stdout:(1/9): libgpg-error-devel-1.31-1.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-01-24T17:29:37.837 INFO:teuthology.orchestra.run.smithi150.stdout:(2/9): libffi-devel-3.1-22.el8.x86_64.rpm 497 kB/s | 29 kB 00:00 2023-01-24T17:29:37.848 INFO:teuthology.orchestra.run.smithi150.stdout:(3/9): xz-devel-5.2.4-3.el8.x86_64.rpm 907 kB/s | 62 kB 00:00 2023-01-24T17:29:37.855 INFO:teuthology.orchestra.run.smithi150.stdout:(4/9): zlib-devel-1.2.11-17.el8.x86_64.rpm 3.1 MB/s | 58 kB 00:00 2023-01-24T17:29:37.866 INFO:teuthology.orchestra.run.smithi150.stdout:(5/9): libgcrypt-devel-1.8.5-4.el8.x86_64.rpm 4.5 MB/s | 149 kB 00:00 2023-01-24T17:29:37.880 INFO:teuthology.orchestra.run.smithi150.stdout:(6/9): libevent-devel-2.1.8-5.el8.x86_64.rpm 3.2 MB/s | 104 kB 00:00 2023-01-24T17:29:37.887 INFO:teuthology.orchestra.run.smithi150.stdout:(7/9): libxslt-devel-1.1.32-6.el8.x86_64.rpm 10 MB/s | 323 kB 00:00 2023-01-24T17:29:37.907 INFO:teuthology.orchestra.run.smithi150.stdout:(8/9): libxml2-devel-2.9.7-9.el8_4.2.x86_64.rpm 26 MB/s | 1.0 MB 00:00 2023-01-24T17:29:37.908 INFO:teuthology.orchestra.run.smithi150.stdout:(9/9): cmake-filesystem-3.18.2-11.el8_4.x86_64. 1.6 MB/s | 44 kB 00:00 2023-01-24T17:29:37.909 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:29:37.909 INFO:teuthology.orchestra.run.smithi150.stdout:Total 14 MB/s | 1.9 MB 00:00 2023-01-24T17:29:37.916 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-01-24T17:29:37.928 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-01-24T17:29:37.929 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-01-24T17:29:38.085 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-01-24T17:29:38.089 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-01-24T17:29:38.484 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-01-24T17:29:38.683 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 1/9 2023-01-24T17:29:38.717 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 1/9 2023-01-24T17:29:38.875 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgcrypt-devel-1.8.5-4.el8.x86_64 2/9 2023-01-24T17:29:38.887 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libgcrypt-devel-1.8.5-4.el8.x86_64 2/9 2023-01-24T17:29:39.074 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cmake-filesystem-3.18.2-11.el8_4.x86_64 3/9 2023-01-24T17:29:39.222 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 4/9 2023-01-24T17:29:39.629 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xz-devel-5.2.4-3.el8.x86_64 5/9 2023-01-24T17:29:39.956 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libxml2-devel-2.9.7-9.el8_4.2.x86_64 6/9 2023-01-24T17:29:40.173 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 7/9 2023-01-24T17:29:40.339 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 8/9 2023-01-24T17:29:40.483 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libffi-devel-3.1-22.el8.x86_64 9/9 2023-01-24T17:29:40.786 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libffi-devel-3.1-22.el8.x86_64 9/9 2023-01-24T17:29:40.786 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libffi-devel-3.1-22.el8.x86_64 1/9 2023-01-24T17:29:40.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xz-devel-5.2.4-3.el8.x86_64 2/9 2023-01-24T17:29:40.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 3/9 2023-01-24T17:29:40.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgcrypt-devel-1.8.5-4.el8.x86_64 4/9 2023-01-24T17:29:40.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 5/9 2023-01-24T17:29:40.787 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 6/9 2023-01-24T17:29:40.788 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 7/9 2023-01-24T17:29:40.788 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libxml2-devel-2.9.7-9.el8_4.2.x86_64 8/9 2023-01-24T17:29:42.143 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cmake-filesystem-3.18.2-11.el8_4.x86_64 9/9 2023-01-24T17:29:42.144 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:05:08 ago on Tue 24 Jan 2023 05:24:34 PM UTC. 2023-01-24T17:29:44.003 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-01-24T17:29:44.103 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:29:44.103 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-01-24T17:29:44.104 INFO:teuthology.orchestra.run.smithi150.stdout: cmake-filesystem-3.18.2-11.el8_4.x86_64 libevent-devel-2.1.8-5.el8.x86_64 2023-01-24T17:29:44.104 INFO:teuthology.orchestra.run.smithi150.stdout: libffi-devel-3.1-22.el8.x86_64 libgcrypt-devel-1.8.5-4.el8.x86_64 2023-01-24T17:29:44.104 INFO:teuthology.orchestra.run.smithi150.stdout: libgpg-error-devel-1.31-1.el8.x86_64 libxml2-devel-2.9.7-9.el8_4.2.x86_64 2023-01-24T17:29:44.104 INFO:teuthology.orchestra.run.smithi150.stdout: libxslt-devel-1.1.32-6.el8.x86_64 xz-devel-5.2.4-3.el8.x86_64 2023-01-24T17:29:44.104 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-01-24T17:29:44.105 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:29:44.105 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-01-24T17:29:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:Already using interpreter /usr/bin/python3.6 2023-01-24T17:29:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:Using base prefix '/usr' 2023-01-24T17:29:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:New python executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python3.6 2023-01-24T17:29:44.580 INFO:teuthology.orchestra.run.smithi150.stdout:Also creating executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python 2023-01-24T17:29:48.582 INFO:teuthology.orchestra.run.smithi150.stdout:Installing setuptools, pip, wheel...done. 2023-01-24T17:29:48.915 INFO:teuthology.orchestra.run.smithi150.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.6/site-packages (21.3.1) 2023-01-24T17:29:50.173 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting setuptools==44.1.0 2023-01-24T17:29:50.244 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2023-01-24T17:29:50.364 INFO:teuthology.orchestra.run.smithi150.stdout:Installing collected packages: setuptools 2023-01-24T17:29:50.365 INFO:teuthology.orchestra.run.smithi150.stdout: Attempting uninstall: setuptools 2023-01-24T17:29:50.365 INFO:teuthology.orchestra.run.smithi150.stdout: Found existing installation: setuptools 59.6.0 2023-01-24T17:29:50.410 INFO:teuthology.orchestra.run.smithi150.stdout: Uninstalling setuptools-59.6.0: 2023-01-24T17:29:50.418 INFO:teuthology.orchestra.run.smithi150.stdout: Successfully uninstalled setuptools-59.6.0 2023-01-24T17:29:50.642 INFO:teuthology.orchestra.run.smithi150.stdout:Successfully installed setuptools-44.1.0 2023-01-24T17:29:51.213 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting PyYAML 2023-01-24T17:29:51.286 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (603 kB) 2023-01-24T17:29:51.430 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting nose>=1.0.0 2023-01-24T17:29:51.448 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2023-01-24T17:29:51.562 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting boto>=2.6.0 2023-01-24T17:29:51.581 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2023-01-24T17:29:52.457 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting boto3>=1.0.0 2023-01-24T17:29:52.517 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2023-01-24T17:29:53.513 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting botocore<1.28.0 2023-01-24T17:29:53.532 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2023-01-24T17:29:53.842 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting munch>=2.0.0 2023-01-24T17:29:53.859 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading munch-2.5.0-py2.py3-none-any.whl (10 kB) 2023-01-24T17:29:54.240 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting gevent>=1.0 2023-01-24T17:29:54.260 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading gevent-22.10.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB) 2023-01-24T17:29:54.498 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting isodate>=0.4.4 2023-01-24T17:29:54.515 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2023-01-24T17:29:54.679 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting requests>=2.23.0 2023-01-24T17:29:54.696 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2023-01-24T17:29:54.890 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting pytz>=2011k 2023-01-24T17:29:54.907 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading pytz-2022.7.1-py2.py3-none-any.whl (499 kB) 2023-01-24T17:29:55.025 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting httplib2 2023-01-24T17:29:55.042 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading httplib2-0.21.0-py3-none-any.whl (96 kB) 2023-01-24T17:29:55.544 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting lxml 2023-01-24T17:29:55.563 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading lxml-4.9.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.6 MB) 2023-01-24T17:29:55.995 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting jmespath<2.0.0,>=0.7.1 2023-01-24T17:29:56.012 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2023-01-24T17:29:56.102 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2023-01-24T17:29:56.119 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2023-01-24T17:29:56.255 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting urllib3<1.27,>=1.25.4 2023-01-24T17:29:56.272 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB) 2023-01-24T17:29:56.371 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting python-dateutil<3.0.0,>=2.1 2023-01-24T17:29:56.389 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-01-24T17:29:56.493 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting six 2023-01-24T17:29:56.510 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-01-24T17:29:56.637 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting zope.event 2023-01-24T17:29:56.654 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB) 2023-01-24T17:29:56.676 INFO:teuthology.orchestra.run.smithi150.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (from gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2023-01-24T17:29:57.054 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting greenlet>=2.0.0 2023-01-24T17:29:57.072 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading greenlet-2.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (513 kB) 2023-01-24T17:29:57.439 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting zope.interface 2023-01-24T17:29:57.458 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253 kB) 2023-01-24T17:29:57.555 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting idna<4,>=2.5 2023-01-24T17:29:57.572 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading idna-3.4-py3-none-any.whl (61 kB) 2023-01-24T17:29:57.652 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting certifi>=2017.4.17 2023-01-24T17:29:57.669 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading certifi-2022.12.7-py3-none-any.whl (155 kB) 2023-01-24T17:29:57.921 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting charset-normalizer~=2.0.0 2023-01-24T17:29:57.938 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2023-01-24T17:29:58.075 INFO:teuthology.orchestra.run.smithi150.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2023-01-24T17:29:58.092 INFO:teuthology.orchestra.run.smithi150.stdout: Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB) 2023-01-24T17:29:58.606 INFO:teuthology.orchestra.run.smithi150.stdout:Installing collected packages: six, urllib3, python-dateutil, jmespath, botocore, zope.interface, zope.event, s3transfer, pyparsing, idna, greenlet, charset-normalizer, certifi, requests, PyYAML, pytz, nose, munch, lxml, isodate, httplib2, gevent, boto3, boto 2023-01-24T17:30:01.888 INFO:teuthology.orchestra.run.smithi150.stdout:Successfully installed PyYAML-6.0 boto-2.49.0 boto3-1.23.10 botocore-1.26.10 certifi-2022.12.7 charset-normalizer-2.0.12 gevent-22.10.2 greenlet-2.0.1 httplib2-0.21.0 idna-3.4 isodate-0.6.1 jmespath-0.10.0 lxml-4.9.2 munch-2.5.0 nose-1.3.7 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2022.7.1 requests-2.27.1 s3transfer-0.5.2 six-1.16.0 urllib3-1.26.14 zope.event-4.6 zope.interface-5.5.2 2023-01-24T17:30:02.188 INFO:teuthology.orchestra.run.smithi150.stdout:running develop 2023-01-24T17:30:02.206 INFO:teuthology.orchestra.run.smithi150.stdout:running egg_info 2023-01-24T17:30:02.207 INFO:teuthology.orchestra.run.smithi150.stdout:creating s3tests.egg-info 2023-01-24T17:30:02.207 INFO:teuthology.orchestra.run.smithi150.stdout:writing s3tests.egg-info/PKG-INFO 2023-01-24T17:30:02.207 INFO:teuthology.orchestra.run.smithi150.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2023-01-24T17:30:02.207 INFO:teuthology.orchestra.run.smithi150.stdout:writing requirements to s3tests.egg-info/requires.txt 2023-01-24T17:30:02.207 INFO:teuthology.orchestra.run.smithi150.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2023-01-24T17:30:02.208 INFO:teuthology.orchestra.run.smithi150.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-01-24T17:30:02.217 INFO:teuthology.orchestra.run.smithi150.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2023-01-24T17:30:02.218 INFO:teuthology.orchestra.run.smithi150.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-01-24T17:30:02.232 INFO:teuthology.orchestra.run.smithi150.stdout:running build_ext 2023-01-24T17:30:02.232 INFO:teuthology.orchestra.run.smithi150.stdout:Creating /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/s3tests.egg-link (link to .) 2023-01-24T17:30:02.232 INFO:teuthology.orchestra.run.smithi150.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2023-01-24T17:30:02.233 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.233 INFO:teuthology.orchestra.run.smithi150.stdout:Installed /home/ubuntu/cephtest/s3-tests 2023-01-24T17:30:02.233 INFO:teuthology.orchestra.run.smithi150.stdout:Processing dependencies for s3tests==0.0.1 2023-01-24T17:30:02.321 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for isodate==0.6.1 2023-01-24T17:30:02.325 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: isodate 0.6.1 2023-01-24T17:30:02.326 INFO:teuthology.orchestra.run.smithi150.stdout:Adding isodate 0.6.1 to easy-install.pth file 2023-01-24T17:30:02.326 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.326 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.326 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for gevent==22.10.2 2023-01-24T17:30:02.326 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: gevent 22.10.2 2023-01-24T17:30:02.327 INFO:teuthology.orchestra.run.smithi150.stdout:Adding gevent 22.10.2 to easy-install.pth file 2023-01-24T17:30:02.327 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.327 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.327 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for munch==2.5.0 2023-01-24T17:30:02.327 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: munch 2.5.0 2023-01-24T17:30:02.328 INFO:teuthology.orchestra.run.smithi150.stdout:Adding munch 2.5.0 to easy-install.pth file 2023-01-24T17:30:02.328 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.328 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.328 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for PyYAML==6.0 2023-01-24T17:30:02.328 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: PyYAML 6.0 2023-01-24T17:30:02.329 INFO:teuthology.orchestra.run.smithi150.stdout:Adding PyYAML 6.0 to easy-install.pth file 2023-01-24T17:30:02.329 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.329 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.329 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for boto3==1.23.10 2023-01-24T17:30:02.329 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: boto3 1.23.10 2023-01-24T17:30:02.329 INFO:teuthology.orchestra.run.smithi150.stdout:Adding boto3 1.23.10 to easy-install.pth file 2023-01-24T17:30:02.330 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.330 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.330 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for boto==2.49.0 2023-01-24T17:30:02.330 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: boto 2.49.0 2023-01-24T17:30:02.330 INFO:teuthology.orchestra.run.smithi150.stdout:Adding boto 2.49.0 to easy-install.pth file 2023-01-24T17:30:02.330 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.331 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.331 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for six==1.16.0 2023-01-24T17:30:02.331 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: six 1.16.0 2023-01-24T17:30:02.331 INFO:teuthology.orchestra.run.smithi150.stdout:Adding six 1.16.0 to easy-install.pth file 2023-01-24T17:30:02.331 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.332 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.335 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for zope.event==4.6 2023-01-24T17:30:02.335 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: zope.event 4.6 2023-01-24T17:30:02.336 INFO:teuthology.orchestra.run.smithi150.stdout:Adding zope.event 4.6 to easy-install.pth file 2023-01-24T17:30:02.336 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.336 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.336 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for setuptools==44.1.0 2023-01-24T17:30:02.336 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: setuptools 44.1.0 2023-01-24T17:30:02.337 INFO:teuthology.orchestra.run.smithi150.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2023-01-24T17:30:02.337 INFO:teuthology.orchestra.run.smithi150.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2023-01-24T17:30:02.337 INFO:teuthology.orchestra.run.smithi150.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2023-01-24T17:30:02.337 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.337 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.338 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for greenlet==2.0.1 2023-01-24T17:30:02.338 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: greenlet 2.0.1 2023-01-24T17:30:02.338 INFO:teuthology.orchestra.run.smithi150.stdout:Adding greenlet 2.0.1 to easy-install.pth file 2023-01-24T17:30:02.338 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.338 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.338 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for zope.interface==5.5.2 2023-01-24T17:30:02.339 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: zope.interface 5.5.2 2023-01-24T17:30:02.339 INFO:teuthology.orchestra.run.smithi150.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2023-01-24T17:30:02.339 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.339 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.339 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for jmespath==0.10.0 2023-01-24T17:30:02.340 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: jmespath 0.10.0 2023-01-24T17:30:02.340 INFO:teuthology.orchestra.run.smithi150.stdout:Adding jmespath 0.10.0 to easy-install.pth file 2023-01-24T17:30:02.340 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.340 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.340 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for botocore==1.26.10 2023-01-24T17:30:02.341 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: botocore 1.26.10 2023-01-24T17:30:02.341 INFO:teuthology.orchestra.run.smithi150.stdout:Adding botocore 1.26.10 to easy-install.pth file 2023-01-24T17:30:02.341 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.341 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.341 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for s3transfer==0.5.2 2023-01-24T17:30:02.341 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: s3transfer 0.5.2 2023-01-24T17:30:02.342 INFO:teuthology.orchestra.run.smithi150.stdout:Adding s3transfer 0.5.2 to easy-install.pth file 2023-01-24T17:30:02.342 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.342 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.342 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for urllib3==1.26.14 2023-01-24T17:30:02.342 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: urllib3 1.26.14 2023-01-24T17:30:02.343 INFO:teuthology.orchestra.run.smithi150.stdout:Adding urllib3 1.26.14 to easy-install.pth file 2023-01-24T17:30:02.343 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.343 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.343 INFO:teuthology.orchestra.run.smithi150.stdout:Searching for python-dateutil==2.8.2 2023-01-24T17:30:02.343 INFO:teuthology.orchestra.run.smithi150.stdout:Best match: python-dateutil 2.8.2 2023-01-24T17:30:02.344 INFO:teuthology.orchestra.run.smithi150.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2023-01-24T17:30:02.344 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:30:02.344 INFO:teuthology.orchestra.run.smithi150.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:30:02.344 INFO:teuthology.orchestra.run.smithi150.stdout:Finished processing dependencies for s3tests==0.0.1 2023-01-24T17:30:02.352 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:30:02.352 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2023-01-24T17:30:02.367 INFO:tasks.s3tests:Configuring boto... 2023-01-24T17:30:02.367 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-01-24T17:30:02.368 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/boto.cfg 2023-01-24T17:30:02.422 DEBUG:teuthology.orchestra.run.smithi150:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain' 2023-01-24T17:30:02.896 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2023-01-24T17:30:02.955 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2023-01-24T17:30:03.017 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2023-01-24T17:30:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2023-01-24T17:30:03.131 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2023-01-24T17:30:03.143 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2023-01-24T17:30:03.198 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2023-01-24T17:30:03.207 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2023-01-24T17:30:03.249 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2023-01-24T17:30:03.290 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2023-01-24T17:30:03.342 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2023-01-24T17:30:03.397 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2023-01-24T17:30:03.448 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2023-01-24T17:30:03.501 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2023-01-24T17:30:03.504 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2023-01-24T17:30:03.549 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2023-01-24T17:30:03.549 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2023-01-24T17:30:03.549 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2023-01-24T17:30:03.549 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2023-01-24T17:30:03.549 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2023-01-24T17:30:03.550 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2023-01-24T17:30:03.550 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2023-01-24T17:30:03.550 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2023-01-24T17:30:03.550 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2023-01-24T17:30:03.551 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2023-01-24T17:30:03.551 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2023-01-24T17:30:03.551 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2023-01-24T17:30:03.551 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2023-01-24T17:30:03.551 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2023-01-24T17:30:03.552 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2023-01-24T17:30:03.552 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2023-01-24T17:30:03.552 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2023-01-24T17:30:03.552 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2023-01-24T17:30:03.552 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-01-24T17:30:03.553 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2023-01-24T17:30:03.553 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2023-01-24T17:30:03.553 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2023-01-24T17:30:03.553 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2023-01-24T17:30:03.553 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2023-01-24T17:30:03.553 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2023-01-24T17:30:03.554 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2023-01-24T17:30:03.554 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2023-01-24T17:30:03.554 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2023-01-24T17:30:03.554 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2023-01-24T17:30:03.554 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2023-01-24T17:30:03.554 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2023-01-24T17:30:03.555 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2023-01-24T17:30:03.555 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2023-01-24T17:30:03.555 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2023-01-24T17:30:03.555 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2023-01-24T17:30:03.555 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2023-01-24T17:30:03.556 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2023-01-24T17:30:03.556 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-01-24T17:30:03.690 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2023-01-24T17:30:03.690 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2023-01-24T17:30:03.690 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2023-01-24T17:30:03.690 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2023-01-24T17:30:03.691 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2023-01-24T17:30:03.691 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2023-01-24T17:30:03.691 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2023-01-24T17:30:03.789 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2023-01-24T17:30:03.843 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2023-01-24T17:30:05.689 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2023-01-24T17:30:05.784 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2023-01-24T17:30:05.854 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2023-01-24T17:30:05.881 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2023-01-24T17:30:05.964 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2023-01-24T17:30:05.964 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2023-01-24T17:30:05.964 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2023-01-24T17:30:05.964 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2023-01-24T17:30:05.965 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2023-01-24T17:30:05.965 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2023-01-24T17:30:05.965 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2023-01-24T17:30:05.965 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2023-01-24T17:30:05.965 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2023-01-24T17:30:05.966 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2023-01-24T17:30:05.966 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2023-01-24T17:30:05.966 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2023-01-24T17:30:05.966 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2023-01-24T17:30:05.966 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_pubilc_errordoc ... SKIP 2023-01-24T17:30:05.967 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2023-01-24T17:30:05.967 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2023-01-24T17:30:05.967 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2023-01-24T17:30:05.967 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2023-01-24T17:30:05.967 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2023-01-24T17:30:05.967 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2023-01-24T17:30:05.968 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2023-01-24T17:30:05.968 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2023-01-24T17:30:05.968 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2023-01-24T17:30:05.968 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2023-01-24T17:30:05.968 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2023-01-24T17:30:05.969 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2023-01-24T17:30:05.969 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2023-01-24T17:30:05.969 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2023-01-24T17:30:05.969 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2023-01-24T17:30:05.970 INFO:teuthology.orchestra.run.smithi150.stderr:SKIP 2023-01-24T17:30:07.663 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests.functional.test_utils.test_generate ... ok 2023-01-24T17:30:07.663 INFO:teuthology.orchestra.run.smithi150.stderr:/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2023-01-24T17:30:07.663 INFO:teuthology.orchestra.run.smithi150.stderr: warnings.warn(warning, PythonDeprecationWarning) 2023-01-24T17:30:07.694 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2023-01-24T17:30:14.758 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2023-01-24T17:30:14.790 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2023-01-24T17:30:14.839 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2023-01-24T17:30:14.885 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2023-01-24T17:30:14.930 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2023-01-24T17:30:14.977 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2023-01-24T17:30:15.030 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2023-01-24T17:30:15.076 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2023-01-24T17:30:15.108 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2023-01-24T17:30:15.137 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2023-01-24T17:30:15.164 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2023-01-24T17:30:15.181 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2023-01-24T17:30:15.201 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2023-01-24T17:30:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2023-01-24T17:30:15.235 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2023-01-24T17:30:15.272 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2023-01-24T17:30:15.309 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2023-01-24T17:30:15.343 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2023-01-24T17:30:15.375 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2023-01-24T17:30:15.400 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2023-01-24T17:30:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2023-01-24T17:30:15.451 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2023-01-24T17:30:15.475 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2023-01-24T17:30:15.491 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2023-01-24T17:30:15.500 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2023-01-24T17:30:15.508 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2023-01-24T17:30:15.517 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2023-01-24T17:30:15.527 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2023-01-24T17:30:15.535 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2023-01-24T17:30:15.616 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2023-01-24T17:30:15.714 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2023-01-24T17:30:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2023-01-24T17:30:15.919 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2023-01-24T17:30:16.012 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2023-01-24T17:30:16.077 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2023-01-24T17:30:16.169 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2023-01-24T17:30:16.234 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2023-01-24T17:30:16.292 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2023-01-24T17:30:16.423 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2023-01-24T17:30:16.556 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2023-01-24T17:30:16.590 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2023-01-24T17:30:16.627 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2023-01-24T17:30:16.691 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2023-01-24T17:30:16.752 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2023-01-24T17:30:16.882 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2023-01-24T17:30:17.033 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2023-01-24T17:30:17.100 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2023-01-24T17:30:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2023-01-24T17:30:17.218 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2023-01-24T17:30:17.289 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2023-01-24T17:30:17.356 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2023-01-24T17:30:17.430 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2023-01-24T17:30:17.488 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2023-01-24T17:30:17.557 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2023-01-24T17:30:17.618 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2023-01-24T17:30:17.680 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2023-01-24T17:30:17.738 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2023-01-24T17:30:17.803 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2023-01-24T17:30:17.879 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2023-01-24T17:30:17.931 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2023-01-24T17:30:17.984 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2023-01-24T17:30:18.040 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2023-01-24T17:30:18.105 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2023-01-24T17:30:25.640 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2023-01-24T17:30:25.700 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2023-01-24T17:30:25.749 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2023-01-24T17:30:25.831 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2023-01-24T17:30:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2023-01-24T17:30:25.952 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2023-01-24T17:30:26.013 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2023-01-24T17:30:26.062 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2023-01-24T17:30:26.119 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2023-01-24T17:30:26.181 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2023-01-24T17:30:26.230 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2023-01-24T17:30:26.283 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2023-01-24T17:30:26.333 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2023-01-24T17:30:26.389 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2023-01-24T17:30:26.448 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2023-01-24T17:30:26.510 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2023-01-24T17:30:26.566 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2023-01-24T17:30:26.639 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2023-01-24T17:30:26.693 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2023-01-24T17:30:26.785 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2023-01-24T17:30:26.844 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2023-01-24T17:30:26.900 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-01-24T17:30:26.963 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-01-24T17:30:27.064 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2023-01-24T17:30:27.169 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2023-01-24T17:30:27.224 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2023-01-24T17:30:27.284 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2023-01-24T17:30:27.348 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2023-01-24T17:30:27.404 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2023-01-24T17:30:27.648 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_account_usage ... ok 2023-01-24T17:30:27.691 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_head_bucket_usage ... ok 2023-01-24T17:30:28.107 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2023-01-24T17:30:28.526 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2023-01-24T17:30:28.589 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2023-01-24T17:30:28.649 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2023-01-24T17:30:28.705 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2023-01-24T17:30:28.761 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2023-01-24T17:30:28.872 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken ... ok 2023-01-24T17:30:28.973 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_both_continuationtoken_startafter ... ok 2023-01-24T17:30:29.033 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2023-01-24T17:30:29.090 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2023-01-24T17:30:29.146 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2023-01-24T17:30:29.202 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2023-01-24T17:30:29.257 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2023-01-24T17:30:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2023-01-24T17:30:29.576 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2023-01-24T17:30:29.890 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2023-01-24T17:30:29.927 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2023-01-24T17:30:29.965 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2023-01-24T17:30:29.993 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2023-01-24T17:30:30.032 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2023-01-24T17:30:30.041 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2023-01-24T17:30:30.052 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2023-01-24T17:30:30.060 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2023-01-24T17:30:30.111 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2023-01-24T17:30:30.269 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2023-01-24T17:30:30.278 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2023-01-24T17:30:30.382 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2023-01-24T17:30:30.411 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_read_not_exist ... ok 2023-01-24T17:30:30.438 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2023-01-24T17:30:30.664 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2023-01-24T17:30:30.897 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2023-01-24T17:30:38.774 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete_key_limit ... ok 2023-01-24T17:30:46.712 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete_key_limit ... ok 2023-01-24T17:30:46.753 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2023-01-24T17:30:46.791 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2023-01-24T17:30:46.833 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2023-01-24T17:30:46.869 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2023-01-24T17:30:47.005 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2023-01-24T17:30:47.049 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2023-01-24T17:30:47.088 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2023-01-24T17:30:47.140 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2023-01-24T17:30:47.192 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2023-01-24T17:30:47.270 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2023-01-24T17:30:47.342 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2023-01-24T17:30:47.381 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2023-01-24T17:30:47.472 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2023-01-24T17:30:47.494 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2023-01-24T17:30:47.522 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2023-01-24T17:30:47.548 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2023-01-24T17:30:47.609 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2023-01-24T17:30:47.645 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2023-01-24T17:30:47.671 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2023-01-24T17:30:47.700 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2023-01-24T17:30:47.741 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2023-01-24T17:30:47.821 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2023-01-24T17:30:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2023-01-24T17:30:47.874 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2023-01-24T17:30:47.898 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2023-01-24T17:30:47.922 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2023-01-24T17:30:47.954 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2023-01-24T17:30:47.984 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2023-01-24T17:30:48.019 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2023-01-24T17:30:48.043 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2023-01-24T17:30:48.067 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2023-01-24T17:30:48.097 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2023-01-24T17:30:48.127 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2023-01-24T17:30:48.154 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2023-01-24T17:30:48.183 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2023-01-24T17:30:48.209 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2023-01-24T17:30:48.233 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2023-01-24T17:30:48.261 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2023-01-24T17:30:48.290 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2023-01-24T17:30:48.321 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2023-01-24T17:30:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2023-01-24T17:30:48.419 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2023-01-24T17:30:48.498 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2023-01-24T17:30:48.532 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2023-01-24T17:30:48.614 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2023-01-24T17:30:48.694 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2023-01-24T17:30:49.737 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2023-01-24T17:30:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2023-01-24T17:30:49.906 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2023-01-24T17:30:50.035 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2023-01-24T17:30:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2023-01-24T17:30:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2023-01-24T17:30:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2023-01-24T17:30:50.601 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2023-01-24T17:30:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2023-01-24T17:30:50.849 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2023-01-24T17:30:51.015 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2023-01-24T17:30:51.049 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2023-01-24T17:30:51.133 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2023-01-24T17:30:51.208 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2023-01-24T17:30:51.256 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2023-01-24T17:30:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2023-01-24T17:30:51.295 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_notexist ... ok 2023-01-24T17:30:51.361 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2023-01-24T17:30:51.396 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2023-01-24T17:30:51.433 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2023-01-24T17:30:51.474 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2023-01-24T17:30:51.512 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2023-01-24T17:30:51.546 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2023-01-24T17:30:51.582 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2023-01-24T17:30:51.705 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2023-01-24T17:30:51.787 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2023-01-24T17:30:51.824 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2023-01-24T17:30:51.855 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2023-01-24T17:30:51.888 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2023-01-24T17:30:51.919 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2023-01-24T17:30:51.958 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2023-01-24T17:30:52.027 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2023-01-24T17:30:52.059 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2023-01-24T17:30:52.095 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2023-01-24T17:30:52.105 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2023-01-24T17:30:52.115 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2023-01-24T17:30:52.127 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2023-01-24T17:30:52.137 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2023-01-24T17:30:52.161 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2023-01-24T17:30:52.222 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2023-01-24T17:30:52.250 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2023-01-24T17:30:52.315 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl ... ok 2023-01-24T17:30:52.379 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl ... ok 2023-01-24T17:30:52.405 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2023-01-24T17:30:52.471 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2023-01-24T17:30:52.586 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2023-01-24T17:30:52.654 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2023-01-24T17:30:52.718 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2023-01-24T17:30:52.796 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2023-01-24T17:30:52.872 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2023-01-24T17:30:52.996 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2023-01-24T17:30:53.071 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2023-01-24T17:30:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2023-01-24T17:30:53.286 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2023-01-24T17:30:53.411 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2023-01-24T17:30:53.505 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2023-01-24T17:30:53.558 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2023-01-24T17:30:53.585 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2023-01-24T17:30:53.722 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2023-01-24T17:30:53.857 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2023-01-24T17:30:53.983 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2023-01-24T17:30:54.109 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2023-01-24T17:30:54.240 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2023-01-24T17:30:54.389 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2023-01-24T17:30:54.506 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2023-01-24T17:30:54.626 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2023-01-24T17:30:54.755 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2023-01-24T17:30:54.872 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2023-01-24T17:30:54.915 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2023-01-24T17:30:55.028 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2023-01-24T17:30:55.106 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2023-01-24T17:30:55.195 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2023-01-24T17:30:55.280 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2023-01-24T17:30:55.321 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_not_exist ... ok 2023-01-24T17:30:55.449 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2023-01-24T17:30:55.667 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2023-01-24T17:30:55.898 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2023-01-24T17:30:56.116 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2023-01-24T17:30:56.326 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2023-01-24T17:30:56.542 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2023-01-24T17:30:56.753 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2023-01-24T17:30:56.970 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2023-01-24T17:30:57.197 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2023-01-24T17:30:57.406 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2023-01-24T17:30:57.580 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2023-01-24T17:30:57.779 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2023-01-24T17:30:57.948 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2023-01-24T17:30:58.138 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2023-01-24T17:30:58.291 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_buckets_list_ctime ... ok 2023-01-24T17:30:58.297 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2023-01-24T17:30:58.306 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2023-01-24T17:30:58.314 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2023-01-24T17:30:58.484 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2023-01-24T17:30:58.655 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2023-01-24T17:30:58.677 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2023-01-24T17:30:58.697 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2023-01-24T17:30:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2023-01-24T17:30:59.432 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2023-01-24T17:30:59.513 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2023-01-24T17:30:59.621 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2023-01-24T17:30:59.754 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2023-01-24T17:30:59.883 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2023-01-24T17:30:59.968 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2023-01-24T17:31:00.059 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2023-01-24T17:31:00.243 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2023-01-24T17:31:00.291 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2023-01-24T17:31:00.407 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2023-01-24T17:31:00.546 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2023-01-24T17:31:00.785 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2023-01-24T17:31:00.977 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2023-01-24T17:31:01.006 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2023-01-24T17:31:01.042 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2023-01-24T17:31:01.651 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2023-01-24T17:31:01.799 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2023-01-24T17:31:02.933 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2023-01-24T17:31:02.977 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2023-01-24T17:31:03.055 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2023-01-24T17:31:03.364 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2023-01-24T17:31:03.452 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2023-01-24T17:31:03.681 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2023-01-24T17:31:04.713 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2023-01-24T17:31:05.486 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2023-01-24T17:31:07.069 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2023-01-24T17:31:11.415 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2023-01-24T17:31:12.290 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2023-01-24T17:31:13.785 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2023-01-24T17:31:13.929 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2023-01-24T17:31:14.284 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2023-01-24T17:31:14.634 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2023-01-24T17:31:14.815 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2023-01-24T17:31:14.891 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2023-01-24T17:31:15.254 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2023-01-24T17:31:15.412 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload_owner ... ok 2023-01-24T17:31:15.503 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2023-01-24T17:31:15.594 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2023-01-24T17:31:15.623 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2023-01-24T17:31:15.752 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2023-01-24T17:31:18.888 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2023-01-24T17:31:21.935 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2023-01-24T17:31:24.981 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2023-01-24T17:31:25.117 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_set_bucket_tagging ... ok 2023-01-24T17:31:25.225 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2023-01-24T17:31:25.410 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2023-01-24T17:31:25.699 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2023-01-24T17:31:25.841 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2023-01-24T17:31:26.108 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2023-01-24T17:31:26.504 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2023-01-24T17:31:26.624 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2023-01-24T17:31:26.896 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2023-01-24T17:31:27.356 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2023-01-24T17:31:27.486 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2023-01-24T17:31:27.665 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2023-01-24T17:31:27.961 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2023-01-24T17:31:28.040 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2023-01-24T17:31:33.869 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2023-01-24T17:31:33.949 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2023-01-24T17:31:34.030 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2023-01-24T17:31:34.112 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2023-01-24T17:31:34.193 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2023-01-24T17:31:34.435 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2023-01-24T17:31:40.573 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2023-01-24T17:31:40.892 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2023-01-24T17:31:41.095 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2023-01-24T17:31:41.400 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2023-01-24T17:31:41.753 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2023-01-24T17:31:44.152 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2023-01-24T17:31:47.295 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2023-01-24T17:31:59.706 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2023-01-24T17:32:01.194 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2023-01-24T17:32:01.905 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2023-01-24T17:32:02.725 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2023-01-24T17:32:03.004 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2023-01-24T17:32:03.316 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2023-01-24T17:32:03.447 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2023-01-24T17:32:03.740 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2023-01-24T17:32:03.917 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2023-01-24T17:32:04.621 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... ok 2023-01-24T17:32:04.653 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2023-01-24T17:32:04.728 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2023-01-24T17:32:04.806 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2023-01-24T17:32:04.839 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2023-01-24T17:32:04.876 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2023-01-24T17:32:04.999 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2023-01-24T17:32:05.042 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2023-01-24T17:32:05.077 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2023-01-24T17:32:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2023-01-24T17:32:05.160 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2023-01-24T17:32:05.194 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2023-01-24T17:32:05.237 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2023-01-24T17:32:05.281 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2023-01-24T17:32:05.317 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_set_invalid_date ... ok 2023-01-24T17:32:05.317 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition ... SKIP 2023-01-24T17:32:05.317 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2023-01-24T17:32:05.318 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2023-01-24T17:32:05.318 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition ... SKIP 2023-01-24T17:32:05.318 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_multiple_transition ... SKIP 2023-01-24T17:32:05.318 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition_large_obj ... SKIP 2023-01-24T17:32:05.400 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2023-01-24T17:32:05.476 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2023-01-24T17:32:05.529 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2023-01-24T17:32:05.615 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2023-01-24T17:32:05.666 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2023-01-24T17:32:05.751 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2023-01-24T17:32:05.828 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2023-01-24T17:32:05.898 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2023-01-24T17:32:05.968 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2023-01-24T17:32:06.044 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2023-01-24T17:32:06.144 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2023-01-24T17:32:07.034 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2023-01-24T17:32:07.627 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2023-01-24T17:32:07.663 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2023-01-24T17:32:07.710 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2023-01-24T17:32:07.787 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2023-01-24T17:32:07.859 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2023-01-24T17:32:07.887 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2023-01-24T17:32:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2023-01-24T17:32:09.297 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2023-01-24T17:32:09.798 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2023-01-24T17:32:09.840 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2023-01-24T17:32:09.925 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2023-01-24T17:32:10.008 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2023-01-24T17:32:10.066 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2023-01-24T17:32:10.147 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2023-01-24T17:32:10.226 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2023-01-24T17:32:10.276 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2023-01-24T17:32:10.335 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2023-01-24T17:32:10.405 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2023-01-24T17:32:10.468 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2023-01-24T17:32:10.632 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2023-01-24T17:32:10.773 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2023-01-24T17:32:10.949 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2023-01-24T17:32:11.086 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2023-01-24T17:32:11.264 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2023-01-24T17:32:11.439 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2023-01-24T17:32:11.607 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2023-01-24T17:32:11.777 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2023-01-24T17:32:11.952 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2023-01-24T17:32:12.181 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2023-01-24T17:32:12.408 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2023-01-24T17:32:12.531 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2023-01-24T17:32:12.576 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2023-01-24T17:32:12.703 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2023-01-24T17:32:12.844 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2023-01-24T17:32:13.021 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2023-01-24T17:32:13.209 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2023-01-24T17:32:13.469 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2023-01-24T17:32:15.181 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2023-01-24T17:32:15.329 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2023-01-24T17:32:15.544 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2023-01-24T17:32:15.719 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2023-01-24T17:32:15.998 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2023-01-24T17:32:16.173 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2023-01-24T17:32:16.264 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2023-01-24T17:32:16.441 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2023-01-24T17:32:16.471 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_s3 ... ok 2023-01-24T17:32:16.501 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_kms ... ok 2023-01-24T17:32:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_s3_kms ... ok 2023-01-24T17:32:16.562 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_bad_enc_kms ... ok 2023-01-24T17:32:16.645 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_s3_noenc ... ok 2023-01-24T17:32:16.727 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_s3_kms ... ok 2023-01-24T17:32:16.815 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_noenc ... ok 2023-01-24T17:32:16.927 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_s3 ... ok 2023-01-24T17:32:17.142 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2023-01-24T17:32:17.215 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2023-01-24T17:32:17.280 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2023-01-24T17:32:17.343 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2023-01-24T17:32:17.413 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2023-01-24T17:32:17.484 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2023-01-24T17:32:17.595 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_mode ... ok 2023-01-24T17:32:17.665 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2023-01-24T17:32:17.744 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2023-01-24T17:32:17.829 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2023-01-24T17:32:17.898 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2023-01-24T17:32:17.984 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2023-01-24T17:32:18.081 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2023-01-24T17:32:18.218 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2023-01-24T17:32:18.357 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2023-01-24T17:32:18.471 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_iso8601 ... ok 2023-01-24T17:32:18.551 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2023-01-24T17:32:18.655 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2023-01-24T17:32:18.773 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2023-01-24T17:32:18.875 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2023-01-24T17:32:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2023-01-24T17:32:19.118 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2023-01-24T17:32:19.230 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2023-01-24T17:32:19.400 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention_and_marker ... ok 2023-01-24T17:32:19.554 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_multi_delete_object_with_retention ... ok 2023-01-24T17:32:19.606 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2023-01-24T17:32:19.682 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2023-01-24T17:32:19.766 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2023-01-24T17:32:19.921 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2023-01-24T17:32:20.000 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2023-01-24T17:32:20.108 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2023-01-24T17:32:20.170 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2023-01-24T17:32:20.253 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2023-01-24T17:32:20.335 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2023-01-24T17:32:20.383 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_with_bypass ... ok 2023-01-24T17:32:20.463 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_without_bypass ... ok 2023-01-24T17:32:20.562 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_compliance ... ok 2023-01-24T17:32:20.691 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifmatch_good ... ok 2023-01-24T17:32:20.830 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifnonematch_failed ... ok 2023-01-24T17:32:20.876 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_user_policy ... ok 2023-01-24T17:32:20.907 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2023-01-24T17:32:20.985 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2023-01-24T17:32:21.063 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2023-01-24T17:32:21.142 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2023-01-24T17:32:21.223 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2023-01-24T17:32:21.301 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2023-01-24T17:32:21.336 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2023-01-24T17:32:21.416 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2023-01-24T17:32:21.621 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2023-01-24T17:32:21.783 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2023-01-24T17:32:21.861 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2023-01-24T17:32:22.055 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2023-01-24T17:32:22.882 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_on_a_bucket_with_policy ... ok 2023-01-24T17:32:22.920 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_bucket_encryption_s3 ... ok 2023-01-24T17:32:22.955 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_put_bucket_encryption_kms ... ok 2023-01-24T17:32:23.037 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_encryption_s3 ... ok 2023-01-24T17:32:23.121 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_encryption_kms ... ok 2023-01-24T17:32:23.213 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_delete_bucket_encryption_s3 ... ok 2023-01-24T17:32:23.298 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_delete_bucket_encryption_kms ... ok 2023-01-24T17:32:23.384 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_upload_1b ... ok 2023-01-24T17:32:23.475 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_upload_1kb ... ok 2023-01-24T17:32:23.543 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_upload_1mb ... ok 2023-01-24T17:32:23.722 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_upload_8mb ... ok 2023-01-24T17:32:23.807 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_upload_1b ... ok 2023-01-24T17:32:23.894 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_upload_1kb ... ok 2023-01-24T17:32:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_upload_1mb ... ok 2023-01-24T17:32:24.147 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_upload_8mb ... ok 2023-01-24T17:32:24.212 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_method_head ... ok 2023-01-24T17:32:25.022 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_multipart_upload ... ok 2023-01-24T17:32:25.117 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_post_object_authenticated_request ... ok 2023-01-24T17:32:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_post_object_authenticated_request ... ok 2023-01-24T17:32:25.301 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_encrypted_upload_1b ... ok 2023-01-24T17:32:25.387 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_encrypted_upload_1kb ... ok 2023-01-24T17:32:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_encrypted_upload_1mb ... ok 2023-01-24T17:32:25.617 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_encrypted_upload_8mb ... ok 2023-01-24T17:32:26.616 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_generate_where_clause ... ok 2023-01-24T17:32:27.582 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_generate_projection ... ok 2023-01-24T17:32:27.641 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_count_operation ... ok 2023-01-24T17:32:28.062 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_column_sum_min_max ... ok 2023-01-24T17:32:28.630 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_nullif_expressions ... ok 2023-01-24T17:32:28.904 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_nulliftrue_expressions ... ok 2023-01-24T17:32:29.117 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_is_not_null_expressions ... ok 2023-01-24T17:32:29.167 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_lowerupper_expressions ... ok 2023-01-24T17:32:29.719 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_in_expressions ... ok 2023-01-24T17:32:30.084 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_true_false_in_expressions ... ok 2023-01-24T17:32:30.629 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_like_expressions ... ok 2023-01-24T17:32:30.951 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_truefalselike_expressions ... ok 2023-01-24T17:32:31.202 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_complex_expressions ... ok 2023-01-24T17:32:31.391 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_alias ... ok 2023-01-24T17:32:31.551 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_alias_cyclic_refernce ... ok 2023-01-24T17:32:31.760 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_datetime ... ok 2023-01-24T17:32:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_true_false_datetime ... ok 2023-01-24T17:32:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_csv_parser ... ok 2023-01-24T17:32:32.339 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_csv_definition ... ok 2023-01-24T17:32:32.579 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_schema_definition ... ok 2023-01-24T17:32:32.807 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_when_then_else_expressions ... ok 2023-01-24T17:32:33.041 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_coalesce_expressions ... ok 2023-01-24T17:32:33.255 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_cast_expressions ... ok 2023-01-24T17:32:33.255 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_version ... ok 2023-01-24T17:32:33.623 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_trim_expressions ... ok 2023-01-24T17:32:34.000 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_truefalse_trim_expressions ... ok 2023-01-24T17:32:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_escape_expressions ... ok 2023-01-24T17:32:34.450 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_case_value_expressions ... ok 2023-01-24T17:32:34.635 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_bool_cast_expressions ... ok 2023-01-24T17:32:47.565 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_progress_expressions ... ok 2023-01-24T17:32:47.565 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_s3select.test_output_serial_expressions ... ok 2023-01-24T17:33:59.926 INFO:teuthology.orchestra.run.smithi150.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2023-01-24T17:33:59.926 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-01-24T17:33:59.926 INFO:teuthology.orchestra.run.smithi150.stderr:---------------------------------------------------------------------- 2023-01-24T17:33:59.927 INFO:teuthology.orchestra.run.smithi150.stderr:Ran 639 tests in 237.287s 2023-01-24T17:33:59.927 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-01-24T17:33:59.927 INFO:teuthology.orchestra.run.smithi150.stderr:OK (SKIP=79) 2023-01-24T17:33:59.993 INFO:teuthology.orchestra.run.smithi150.stderr:Exception ignored in: .generate at 0x7f45d445f150> 2023-01-24T17:33:59.994 INFO:teuthology.orchestra.run.smithi150.stderr:RuntimeError: generator ignored GeneratorExit 2023-01-24T17:34:00.004 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2023-01-24T17:34:00.005 DEBUG:teuthology.orchestra.run.smithi150:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2023-01-24T17:34:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T17:34:00.074 INFO:tasks.s3tests:Cleaning up boto... 2023-01-24T17:34:00.075 DEBUG:teuthology.orchestra.run.smithi150:> rm /home/ubuntu/cephtest/boto.cfg 2023-01-24T17:34:00.089 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2023-01-24T17:34:00.169 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:34:00.169 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:34:00.174 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.172+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.173+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.173+0000 7f37ead01680 20 realm 2023-01-24T17:34:00.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.173+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.177 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.173+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.177 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.173+0000 7f37ead01680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:34:00.177 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.173+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.177 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.173+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.177 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.173+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.178 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.174+0000 7f37ead01680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.178 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.174+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.178 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.174+0000 7f37ead01680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:34:00.178 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.174+0000 7f37ead01680 20 searching for the correct realm 2023-01-24T17:34:00.186 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.182+0000 7f37ead01680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:34:00.186 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.182+0000 7f37ead01680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:34:00.186 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.182+0000 7f37ead01680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:34:00.187 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.182+0000 7f37ead01680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:34:00.187 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.182+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.187 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.182+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.187 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.182+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.182+0000 7f37ead01680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.182+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.189 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.183+0000 7f37ead01680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:34:00.189 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.183+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.189 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.183+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.189 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.183+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.189 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.183+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.190 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.183+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.190 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.183+0000 7f37ead01680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.193 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:34:00.193 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:34:00.193 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:34:00.194 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:34:00.194 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.194 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.194 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.194 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:34:00.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:34:00.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.190+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 20 zone default found 2023-01-24T17:34:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 4 Realm: () 2023-01-24T17:34:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 4 ZoneGroup: default (ea8e6aae-de17-4da7-a2fe-dcef04325757) 2023-01-24T17:34:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 4 Zone: default (60550c17-6c7c-4ff3-b915-efe39561b124) 2023-01-24T17:34:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:34:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 20 zonegroup default 2023-01-24T17:34:00.199 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.199 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.199 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.199 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.199 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.199 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.200 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 20 started sync module instance, tier type = 2023-01-24T17:34:00.200 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.191+0000 7f37ead01680 20 started zone id=60550c17-6c7c-4ff3-b915-efe39561b124 (name=default) with tier type = 2023-01-24T17:34:00.214 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.212+0000 7f37ead01680 20 add_watcher() i=0 2023-01-24T17:34:00.215 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.212+0000 7f37ead01680 20 add_watcher() i=1 2023-01-24T17:34:00.215 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.212+0000 7f37ead01680 20 add_watcher() i=2 2023-01-24T17:34:00.215 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.212+0000 7f37ead01680 20 add_watcher() i=3 2023-01-24T17:34:00.215 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.212+0000 7f37ead01680 20 add_watcher() i=4 2023-01-24T17:34:00.215 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.212+0000 7f37ead01680 20 add_watcher() i=5 2023-01-24T17:34:00.215 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.212+0000 7f37ead01680 20 add_watcher() i=6 2023-01-24T17:34:00.216 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.213+0000 7f37ead01680 20 add_watcher() i=7 2023-01-24T17:34:00.216 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.213+0000 7f37ead01680 2 all 8 watchers are set, enabling cache 2023-01-24T17:34:00.219 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.216+0000 7f37af7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:34:00.219 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.216+0000 7f37ead01680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:34:00.219 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.217+0000 7f37ead01680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:34:00.271 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f371e7fc700 20 reqs_thread_entry: start 2023-01-24T17:34:00.272 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f370ffff700 20 reqs_thread_entry: start 2023-01-24T17:34:00.272 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f37ead01680 20 init_complete bucket index max shards: 11 2023-01-24T17:34:00.272 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f37ead01680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:34:00.272 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f370e7fc700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:34:30 2023 2023-01-24T17:34:00.273 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-01-24T17:34:00.273 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f370dffb700 20 reqs_thread_entry: start 2023-01-24T17:34:00.273 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f37ead01680 20 get_system_obj_state: rctx=0x7ffec1ec17f8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55f30d6e5940 s->prefetch_data=0 2023-01-24T17:34:00.273 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f37ead01680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-01-24T17:34:00.273 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f37ead01680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-01-24T17:34:00.274 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f37ead01680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:34:00.274 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f37ead01680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.274 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f37ead01680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:34:00.274 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f37ead01680 20 Read xattr: user.rgw.idtag 2023-01-24T17:34:00.274 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f37ead01680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:34:00.275 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.268+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.275 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.269+0000 7f37ead01680 20 rados_obj.operate() r=0 bl.length=469 2023-01-24T17:34:00.275 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.269+0000 7f37ead01680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-01-24T17:34:00.275 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.269+0000 7f37ead01680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:34:00.275 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.269+0000 7f37ead01680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.278 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.269+0000 7f37ead01680 20 get_system_obj_state: rctx=0x7ffec1ec14b0 obj=default.rgw.meta:root:.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 state=0x55f30d6e5940 s->prefetch_data=0 2023-01-24T17:34:00.278 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.269+0000 7f37ead01680 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 : miss 2023-01-24T17:34:00.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 info.flags=0x16 2023-01-24T17:34:00.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 10 adding default.rgw.meta+root+.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 to cache LRU end 2023-01-24T17:34:00.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:34:00.279 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:34:00.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 20 Read xattr: user.rgw.acl 2023-01-24T17:34:00.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 20 Read xattr: user.rgw.idtag 2023-01-24T17:34:00.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:34:00.280 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.281 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 20 rados_obj.operate() r=0 bl.length=364 2023-01-24T17:34:00.281 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 info.flags=0x13 2023-01-24T17:34:00.281 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 10 moving default.rgw.meta+root+.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 to cache LRU end 2023-01-24T17:34:00.281 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.281 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:34:00.282 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 2023-01-24T17:34:00.282 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.270+0000 7f37ead01680 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :test[60550c17-6c7c-4ff3-b915-efe39561b124.4215.538]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-01-24T17:34:00.282 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.271+0000 7f37ead01680 10 cls_bucket_list_unordered: got 68B87D 2023-01-24T17:34:00.282 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.271+0000 7f37ead01680 10 cls_bucket_list_unordered: got B4F4DC 2023-01-24T17:34:00.282 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.271+0000 7f37ead01680 10 cls_bucket_list_unordered: got 3F6E33 2023-01-24T17:34:00.282 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.271+0000 7f37ead01680 10 cls_bucket_list_unordered: got 509B86 2023-01-24T17:34:00.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.271+0000 7f37ead01680 10 cls_bucket_list_unordered: got 55B489 2023-01-24T17:34:00.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.271+0000 7f37ead01680 10 cls_bucket_list_unordered: got 78D392 2023-01-24T17:34:00.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.271+0000 7f37ead01680 10 cls_bucket_list_unordered: got 8FC286 2023-01-24T17:34:00.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.271+0000 7f37ead01680 10 cls_bucket_list_unordered: got E14845 2023-01-24T17:34:00.283 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.271+0000 7f37ead01680 10 cls_bucket_list_unordered: got 41CDDB 2023-01-24T17:34:00.284 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.271+0000 7f37ead01680 10 cls_bucket_list_unordered: got 67FB24 2023-01-24T17:34:00.284 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.271+0000 7f37ead01680 10 cls_bucket_list_unordered: got 970DE6 2023-01-24T17:34:00.284 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.271+0000 7f37ead01680 10 cls_bucket_list_unordered: got A58E54 2023-01-24T17:34:00.284 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.272+0000 7f37ead01680 10 cls_bucket_list_unordered: got 0C0BE5 2023-01-24T17:34:00.285 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.272+0000 7f37ead01680 10 cls_bucket_list_unordered: got 3DAA7E 2023-01-24T17:34:00.285 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.272+0000 7f37ead01680 10 cls_bucket_list_unordered: got 7B4034 2023-01-24T17:34:00.285 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.272+0000 7f37ead01680 10 cls_bucket_list_unordered: got 978990 2023-01-24T17:34:00.285 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.272+0000 7f37ead01680 10 cls_bucket_list_unordered: got 345EF8 2023-01-24T17:34:00.286 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.273+0000 7f37ead01680 10 cls_bucket_list_unordered: got 611422 2023-01-24T17:34:00.286 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.273+0000 7f37ead01680 10 cls_bucket_list_unordered: got F57A59 2023-01-24T17:34:00.287 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.273+0000 7f37ead01680 10 cls_bucket_list_unordered: got 0EF2D0 2023-01-24T17:34:00.287 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.273+0000 7f37ead01680 10 cls_bucket_list_unordered: got 2287C3 2023-01-24T17:34:00.287 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.273+0000 7f37ead01680 10 cls_bucket_list_unordered: got 9B21E7 2023-01-24T17:34:00.288 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.273+0000 7f37ead01680 10 cls_bucket_list_unordered: got ABCE31 2023-01-24T17:34:00.288 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.273+0000 7f37ead01680 10 cls_bucket_list_unordered: got EDA328 2023-01-24T17:34:00.288 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.274+0000 7f37ead01680 10 cls_bucket_list_unordered: got 74D7E5 2023-01-24T17:34:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.274+0000 7f37ead01680 10 cls_bucket_list_unordered: got C9181B 2023-01-24T17:34:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.274+0000 7f37ead01680 10 cls_bucket_list_unordered: got E12677 2023-01-24T17:34:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.274+0000 7f37ead01680 10 cls_bucket_list_unordered: got F705F2 2023-01-24T17:34:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.274+0000 7f37ead01680 10 cls_bucket_list_unordered: got 839486 2023-01-24T17:34:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.274+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:68B87D state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.275+0000 7f37ead01680 10 manifest: total_size = 38 2023-01-24T17:34:00.291 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.275+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.6403881437100653155 2023-01-24T17:34:00.291 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.275+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:68B87D state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.291 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.275+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.292 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.275+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.0 2023-01-24T17:34:00.296 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : miss 2023-01-24T17:34:00.297 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.297 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.297 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 10 cache put: name=default.rgw.log++bucket.sync-source-hints.test info.flags=0x0 2023-01-24T17:34:00.297 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 10 adding default.rgw.log++bucket.sync-source-hints.test to cache LRU end 2023-01-24T17:34:00.297 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : miss 2023-01-24T17:34:00.298 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.298 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.298 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 10 cache put: name=default.rgw.log++bucket.sync-target-hints.test info.flags=0x0 2023-01-24T17:34:00.298 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 10 adding default.rgw.log++bucket.sync-target-hints.test to cache LRU end 2023-01-24T17:34:00.298 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.277+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:B4F4DC state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.278+0000 7f37ead01680 10 manifest: total_size = 5 2023-01-24T17:34:00.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.278+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.7599841915808297023 2023-01-24T17:34:00.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.278+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:B4F4DC state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.278+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.278+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.0 2023-01-24T17:34:00.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.280+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.280+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.280+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.280+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.280+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.280+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:3F6E33 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.280+0000 7f37ead01680 10 manifest: total_size = 399275 2023-01-24T17:34:00.302 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.280+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.4360358224966244107 2023-01-24T17:34:00.302 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.280+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:3F6E33 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.302 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.280+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.302 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.280+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.1 2023-01-24T17:34:00.302 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.283+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.303 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.283+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.303 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.283+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.303 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.283+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.303 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.283+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.303 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.283+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:509B86 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.303 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.283+0000 7f37ead01680 10 manifest: total_size = 398842 2023-01-24T17:34:00.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.283+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.16446180040464984316 2023-01-24T17:34:00.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.283+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:509B86 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.283+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.283+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.1 2023-01-24T17:34:00.304 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.286+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.286+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.286+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.286+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.286+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.286+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:55B489 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.305 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.286+0000 7f37ead01680 10 manifest: total_size = 180000 2023-01-24T17:34:00.306 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.286+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.7810181431133300378 2023-01-24T17:34:00.306 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.286+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:55B489 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.306 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.286+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.306 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.286+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.1 2023-01-24T17:34:00.306 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.289+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.306 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.289+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.307 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.289+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.307 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.289+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.307 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.289+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.307 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.289+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:78D392 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.289+0000 7f37ead01680 10 manifest: total_size = 5 2023-01-24T17:34:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.289+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.15446636642664664639 2023-01-24T17:34:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.289+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:78D392 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.309 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.289+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.309 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.289+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.1 2023-01-24T17:34:00.309 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.291+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.309 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.291+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.309 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.291+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.310 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.291+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.310 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.291+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.310 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.291+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:8FC286 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.310 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.292+0000 7f37ead01680 10 manifest: total_size = 642232 2023-01-24T17:34:00.310 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.292+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.2273379248084994135 2023-01-24T17:34:00.311 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.292+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:8FC286 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.311 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.292+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.311 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.292+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.1 2023-01-24T17:34:00.313 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.298+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.313 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.298+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.313 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.298+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.298+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.298+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.298+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:E14845 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.298+0000 7f37ead01680 10 manifest: total_size = 398906 2023-01-24T17:34:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.298+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.5858224981862364174 2023-01-24T17:34:00.316 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.298+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:E14845 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.316 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.298+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.316 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.298+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.1 2023-01-24T17:34:00.316 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.304+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.317 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.304+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.317 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.304+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.317 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.304+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.318 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.304+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.318 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.304+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:41CDDB state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.318 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.305+0000 7f37ead01680 10 manifest: total_size = 399056 2023-01-24T17:34:00.319 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.305+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.12142216453013234520 2023-01-24T17:34:00.319 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.305+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:41CDDB state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.319 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.305+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.319 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.305+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.2 2023-01-24T17:34:00.320 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.312+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.321 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.312+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.321 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.312+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.321 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.312+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.321 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.312+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.321 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.312+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:67FB24 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.322 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.312+0000 7f37ead01680 10 manifest: total_size = 577346 2023-01-24T17:34:00.322 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.312+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.15422257300538019474 2023-01-24T17:34:00.322 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.312+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:67FB24 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.322 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.312+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.322 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.312+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.2 2023-01-24T17:34:00.323 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.320+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.324 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.320+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.324 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.320+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.324 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.320+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.324 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.320+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.324 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.320+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:970DE6 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.325 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.320+0000 7f37ead01680 10 manifest: total_size = 399088 2023-01-24T17:34:00.325 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.320+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.12009895403662946504 2023-01-24T17:34:00.325 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.320+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:970DE6 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.325 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.320+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.325 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.320+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.2 2023-01-24T17:34:00.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.328+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.328+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.331 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.328+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.332 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.328+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.332 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.328+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.332 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.328+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:A58E54 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.332 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.328+0000 7f37ead01680 10 manifest: total_size = 117 2023-01-24T17:34:00.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.328+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.16389861501776525706 2023-01-24T17:34:00.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.328+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:A58E54 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.328+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.328+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.2 2023-01-24T17:34:00.337 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.334+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.334+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.334+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.334+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.334+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.334+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:0C0BE5 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.335+0000 7f37ead01680 10 manifest: total_size = 399255 2023-01-24T17:34:00.339 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.335+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.9750729163986959669 2023-01-24T17:34:00.339 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.335+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:0C0BE5 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.339 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.335+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.339 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.335+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.3 2023-01-24T17:34:00.342 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.340+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.343 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.340+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.343 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.340+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.343 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.340+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.343 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.340+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.344 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.340+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:3DAA7E state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.344 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.340+0000 7f37ead01680 10 manifest: total_size = 49268 2023-01-24T17:34:00.344 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.340+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.2546155232386859865 2023-01-24T17:34:00.345 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.340+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:3DAA7E state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.345 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.340+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.345 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.340+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.3 2023-01-24T17:34:00.347 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.345+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.347 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.345+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.348 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.345+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.348 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.345+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.348 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.345+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.348 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.345+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:7B4034 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.349 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.345+0000 7f37ead01680 10 manifest: total_size = 398906 2023-01-24T17:34:00.349 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.345+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.10974092596739243072 2023-01-24T17:34:00.350 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.345+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:7B4034 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.350 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.345+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.350 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.345+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.4 2023-01-24T17:34:00.354 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.351+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.354 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.351+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.354 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.351+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.354 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.351+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.354 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.351+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.355 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.351+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:978990 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.355 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.351+0000 7f37ead01680 10 manifest: total_size = 399205 2023-01-24T17:34:00.355 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.351+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.14974370987494648777 2023-01-24T17:34:00.355 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.351+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:978990 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.355 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.351+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.356 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.351+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.4 2023-01-24T17:34:00.360 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.357+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.360 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.357+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.361 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.357+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.361 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.357+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.361 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.357+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.361 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.357+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:345EF8 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.361 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.357+0000 7f37ead01680 10 manifest: total_size = 398897 2023-01-24T17:34:00.362 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.357+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.7406791810666414178 2023-01-24T17:34:00.362 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.357+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:345EF8 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.362 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.357+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.362 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.357+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.5 2023-01-24T17:34:00.365 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.362+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.365 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.362+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.365 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.362+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.362+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.362+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.362+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:611422 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.366 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.363+0000 7f37ead01680 10 manifest: total_size = 39911153 2023-01-24T17:34:00.367 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.363+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.2871238853748360844 2023-01-24T17:34:00.367 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.363+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:611422 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.367 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.363+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.367 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.363+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.6 2023-01-24T17:34:00.371 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.371 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:34:00.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=0 2023-01-24T17:34:00.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:34:00.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=0 2023-01-24T17:34:00.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:34:00.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): result: ofs=16777216 stripe_ofs=16777216 part_ofs=0 rule->part_size=0 2023-01-24T17:34:00.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:34:00.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): result: ofs=20971520 stripe_ofs=20971520 part_ofs=0 rule->part_size=0 2023-01-24T17:34:00.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:34:00.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): result: ofs=25165824 stripe_ofs=25165824 part_ofs=0 rule->part_size=0 2023-01-24T17:34:00.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:34:00.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): result: ofs=29360128 stripe_ofs=29360128 part_ofs=0 rule->part_size=0 2023-01-24T17:34:00.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:34:00.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): result: ofs=33554432 stripe_ofs=33554432 part_ofs=0 rule->part_size=0 2023-01-24T17:34:00.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:34:00.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): result: ofs=37748736 stripe_ofs=37748736 part_ofs=0 rule->part_size=0 2023-01-24T17:34:00.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:34:00.375 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 RGWObjManifest::operator++(): result: ofs=39911153 stripe_ofs=39911153 part_ofs=0 rule->part_size=0 2023-01-24T17:34:00.376 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.368+0000 7f37ead01680 20 garbage collection: RGWGC::send_chain - on object name: gc.23tag is: 60550c17-6c7c-4ff3-b915-efe39561b124.4213.2871238853748360844 2023-01-24T17:34:00.376 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.373+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:F57A59 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.376 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.373+0000 7f37ead01680 10 manifest: total_size = 399156 2023-01-24T17:34:00.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.373+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.9814882207386022088 2023-01-24T17:34:00.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.373+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:F57A59 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.373+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.373+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.6 2023-01-24T17:34:00.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.377+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.377+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.377+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.377+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.377+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.377+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:0EF2D0 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.378+0000 7f37ead01680 10 manifest: total_size = 399296 2023-01-24T17:34:00.382 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.378+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.16281882037199918209 2023-01-24T17:34:00.382 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.378+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:0EF2D0 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.382 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.378+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.382 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.378+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.8 2023-01-24T17:34:00.387 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.384+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.387 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.384+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.387 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.384+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.387 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.384+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.387 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.384+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.387 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.384+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:2287C3 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.388 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.384+0000 7f37ead01680 10 manifest: total_size = 399178 2023-01-24T17:34:00.388 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.384+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.11045647760665184115 2023-01-24T17:34:00.388 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.384+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:2287C3 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.388 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.384+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.389 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.384+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.8 2023-01-24T17:34:00.393 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.390+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.393 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.390+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.393 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.390+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.393 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.390+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.394 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.390+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.394 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.390+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:9B21E7 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.394 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.390+0000 7f37ead01680 10 manifest: total_size = 359852 2023-01-24T17:34:00.394 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.390+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.10287365964849561889 2023-01-24T17:34:00.394 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.390+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:9B21E7 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.395 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.390+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.395 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.390+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.8 2023-01-24T17:34:00.400 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.397+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.400 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.397+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.400 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.397+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.401 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.397+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.401 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.397+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.401 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.397+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:ABCE31 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.401 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.397+0000 7f37ead01680 10 manifest: total_size = 589872 2023-01-24T17:34:00.401 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.397+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.10847676653939948356 2023-01-24T17:34:00.401 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.397+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:ABCE31 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.402 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.397+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.402 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.397+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.8 2023-01-24T17:34:00.406 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.404+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.406 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.404+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.404+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.404+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.404+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.407 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.404+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:EDA328 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.404+0000 7f37ead01680 10 manifest: total_size = 180000 2023-01-24T17:34:00.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.404+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.716349983290332251 2023-01-24T17:34:00.408 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.404+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:EDA328 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.404+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.409 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.404+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.8 2023-01-24T17:34:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.410+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.410+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.410+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.410+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.414 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.410+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.414 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.410+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:74D7E5 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.414 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.410+0000 7f37ead01680 10 manifest: total_size = 399074 2023-01-24T17:34:00.414 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.410+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.3845956270130286351 2023-01-24T17:34:00.414 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.410+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:74D7E5 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.415 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.410+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.415 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.410+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.9 2023-01-24T17:34:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.417+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.417+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.417+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.417+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.421 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.417+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.421 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.417+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:C9181B state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.421 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.417+0000 7f37ead01680 10 manifest: total_size = 399132 2023-01-24T17:34:00.421 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.417+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.14797399544665437342 2023-01-24T17:34:00.421 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.417+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:C9181B state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.422 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.417+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.422 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.417+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.9 2023-01-24T17:34:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.423+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.423+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.423+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.423+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.423+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.423+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:E12677 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.424+0000 7f37ead01680 10 manifest: total_size = 574126 2023-01-24T17:34:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.424+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.5776260702850072257 2023-01-24T17:34:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.424+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:E12677 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.424+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.424+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.9 2023-01-24T17:34:00.430 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.428+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.431 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.428+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.431 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.428+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.431 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.428+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.431 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.428+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.431 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.428+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:F705F2 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.432 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.428+0000 7f37ead01680 10 manifest: total_size = 644760 2023-01-24T17:34:00.432 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.428+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.6089640066634420397 2023-01-24T17:34:00.432 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.428+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:F705F2 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.433 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.428+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.433 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.428+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.9 2023-01-24T17:34:00.433 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.430+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.434 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.430+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.434 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.430+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.434 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.430+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.434 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.430+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.434 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.431+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:839486 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.435 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.431+0000 7f37ead01680 10 manifest: total_size = 399045 2023-01-24T17:34:00.435 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.431+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.1821941029422138838 2023-01-24T17:34:00.435 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.431+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=test:839486 state=0x55f30d97ba98 s->prefetch_data=0 2023-01-24T17:34:00.435 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.431+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d97ba98 2023-01-24T17:34:00.435 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.431+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.538.10 2023-01-24T17:34:00.436 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.433+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:34:00.436 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.433+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:34:00.437 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.433+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.437 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.433+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.437 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.433+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.437 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.433+0000 7f37ead01680 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-01-24T17:34:00.437 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.433+0000 7f37ead01680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :test[60550c17-6c7c-4ff3-b915-efe39561b124.4215.538]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-01-24T17:34:00.438 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.433+0000 7f37ead01680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-01-24T17:34:00.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-01-24T17:34:00.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-01-24T17:34:00.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-01-24T17:34:00.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-01-24T17:34:00.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test[60550c17-6c7c-4ff3-b915-efe39561b124.4215.538])"; uploads.size()=0; is_truncated=0 2023-01-24T17:34:00.440 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:test[60550c17-6c7c-4ff3-b915-efe39561b124.4215.538]) 2023-01-24T17:34:00.440 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 20 lifecycle: get_system_obj_state: rctx=0x7ffec1ec0f88 obj=default.rgw.meta:root:test state=0x55f30d6e5940 s->prefetch_data=0 2023-01-24T17:34:00.440 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 10 lifecycle: cache get: name=default.rgw.meta+root+test : miss 2023-01-24T17:34:00.440 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-01-24T17:34:00.440 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 10 lifecycle: adding default.rgw.meta+root+test to cache LRU end 2023-01-24T17:34:00.441 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.441 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:34:00.441 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 20 lifecycle: Read xattr: user.rgw.idtag 2023-01-24T17:34:00.441 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 10 lifecycle: cache get: name=default.rgw.meta+root+test : type miss (requested=0x13, cached=0x16) 2023-01-24T17:34:00.441 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.435+0000 7f37ead01680 20 lifecycle: rados->read ofs=0 len=0 2023-01-24T17:34:00.442 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.436+0000 7f37ead01680 20 lifecycle: rados_obj.operate() r=0 bl.length=186 2023-01-24T17:34:00.442 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.436+0000 7f37ead01680 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x13 2023-01-24T17:34:00.442 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.436+0000 7f37ead01680 10 lifecycle: moving default.rgw.meta+root+test to cache LRU end 2023-01-24T17:34:00.442 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.436+0000 7f37ead01680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.443 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.437+0000 7f37ead01680 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 info.flags=0x17 2023-01-24T17:34:00.443 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.437+0000 7f37ead01680 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 to cache LRU end 2023-01-24T17:34:00.443 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.437+0000 7f37ead01680 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:34:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.437+0000 7f37ead01680 10 lifecycle: distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538, ofs0, ns] 2023-01-24T17:34:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.437+0000 7f37d6ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856066 cookie 94502392353104 notifier 4255 bl.length()=800 2023-01-24T17:34:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.437+0000 7f37d6ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 info.flags=0x17 2023-01-24T17:34:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.437+0000 7f37d6ffd700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 to cache LRU end 2023-01-24T17:34:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.437+0000 7f37d6ffd700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:34:00.445 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.438+0000 7f37ead01680 20 lifecycle: handle_bi_update(): bucket=:test[60550c17-6c7c-4ff3-b915-efe39561b124.4215.538]): orig_sources= new_sources= 2023-01-24T17:34:00.445 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.438+0000 7f37ead01680 20 lifecycle: handle_bi_update(): bucket=:test[60550c17-6c7c-4ff3-b915-efe39561b124.4215.538]): potential sources added=[] removed=[] 2023-01-24T17:34:00.445 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.438+0000 7f37ead01680 20 lifecycle: handle_bi_update(): bucket=:test[60550c17-6c7c-4ff3-b915-efe39561b124.4215.538]): orig_dests= new_dests= 2023-01-24T17:34:00.445 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.438+0000 7f37ead01680 20 lifecycle: handle_bi_update(): bucket=:test[60550c17-6c7c-4ff3-b915-efe39561b124.4215.538]): potential dests added=[] removed=[] 2023-01-24T17:34:00.446 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.443+0000 7f37ead01680 20 get_system_obj_state: rctx=0x7ffec1ec09e8 obj=default.rgw.meta:root:test state=0x55f30d6e5940 s->prefetch_data=0 2023-01-24T17:34:00.446 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.443+0000 7f37ead01680 10 cache get: name=default.rgw.meta+root+test : type miss (requested=0x16, cached=0x13) 2023-01-24T17:34:00.446 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.443+0000 7f37ead01680 10 cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-01-24T17:34:00.446 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.443+0000 7f37ead01680 10 moving default.rgw.meta+root+test to cache LRU end 2023-01-24T17:34:00.447 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.443+0000 7f37ead01680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.447 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.443+0000 7f37ead01680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:34:00.447 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.443+0000 7f37ead01680 10 cache get: name=default.rgw.meta+root+test : hit (requested=0x11, cached=0x17) 2023-01-24T17:34:00.447 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.443+0000 7f37ead01680 10 removing default.rgw.meta+root+test from cache 2023-01-24T17:34:00.447 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.443+0000 7f37ead01680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:test, ofs0, ns] 2023-01-24T17:34:00.448 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.443+0000 7f37d6ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856067 cookie 94502392368576 notifier 4255 bl.length()=148 2023-01-24T17:34:00.448 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.445+0000 7f37ead01680 10 removing default.rgw.meta+root+.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 from cache 2023-01-24T17:34:00.449 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.445+0000 7f37ead01680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test:60550c17-6c7c-4ff3-b915-efe39561b124.4215.538, ofs0, ns] 2023-01-24T17:34:00.449 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.445+0000 7f37d77fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856068 cookie 94502392353104 notifier 4255 bl.length()=207 2023-01-24T17:34:00.456 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.453+0000 7f37ead01680 10 removing default.rgw.log++pubsub..bucket.test/60550c17-6c7c-4ff3-b915-efe39561b124.4215.538 from cache 2023-01-24T17:34:00.456 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.453+0000 7f37ead01680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test/60550c17-6c7c-4ff3-b915-efe39561b124.4215.538, ofs0, ns] 2023-01-24T17:34:00.457 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.454+0000 7f37d6ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856362 cookie 94502392349792 notifier 4255 bl.length()=204 2023-01-24T17:34:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.456+0000 7f37ead01680 20 get_system_obj_state: rctx=0x7ffec1ec14b0 obj=default.rgw.meta:root:.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 state=0x55f30d6e5940 s->prefetch_data=0 2023-01-24T17:34:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.456+0000 7f37ead01680 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 : miss 2023-01-24T17:34:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 info.flags=0x16 2023-01-24T17:34:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 10 adding default.rgw.meta+root+.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 to cache LRU end 2023-01-24T17:34:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:34:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:34:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 20 Read xattr: user.rgw.acl 2023-01-24T17:34:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 20 Read xattr: user.rgw.idtag 2023-01-24T17:34:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:34:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.462 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 20 rados_obj.operate() r=0 bl.length=369 2023-01-24T17:34:00.462 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 info.flags=0x13 2023-01-24T17:34:00.462 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 10 moving default.rgw.meta+root+.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 to cache LRU end 2023-01-24T17:34:00.463 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.463 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:34:00.463 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 2023-01-24T17:34:00.463 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.457+0000 7f37ead01680 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :testbuck2[60550c17-6c7c-4ff3-b915-efe39561b124.4215.539]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-01-24T17:34:00.464 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.460+0000 7f37ead01680 10 cls_bucket_list_unordered: got 06DDF2 2023-01-24T17:34:00.464 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.461+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=testbuck2:06DDF2 state=0x55f30d8fc838 s->prefetch_data=0 2023-01-24T17:34:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.461+0000 7f37ead01680 10 manifest: total_size = 399156 2023-01-24T17:34:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.461+0000 7f37ead01680 20 get_obj_state: setting s->obj_tag to 60550c17-6c7c-4ff3-b915-efe39561b124.4213.18368395131037859843 2023-01-24T17:34:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.461+0000 7f37ead01680 20 get_obj_state: rctx=0x7ffec1ec14c0 obj=testbuck2:06DDF2 state=0x55f30d8fc838 s->prefetch_data=0 2023-01-24T17:34:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.461+0000 7f37ead01680 20 prepare_atomic_modification: state is not atomic. state=0x55f30d8fc838 2023-01-24T17:34:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.461+0000 7f37ead01680 20 bucket index object: default.rgw.buckets.index:.dir.60550c17-6c7c-4ff3-b915-efe39561b124.4215.539.9 2023-01-24T17:34:00.468 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.testbuck2 : miss 2023-01-24T17:34:00.469 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.469 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.469 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 10 cache put: name=default.rgw.log++bucket.sync-source-hints.testbuck2 info.flags=0x0 2023-01-24T17:34:00.469 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 10 adding default.rgw.log++bucket.sync-source-hints.testbuck2 to cache LRU end 2023-01-24T17:34:00.469 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.testbuck2 : miss 2023-01-24T17:34:00.470 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.470 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.470 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 10 cache put: name=default.rgw.log++bucket.sync-target-hints.testbuck2 info.flags=0x0 2023-01-24T17:34:00.470 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 10 adding default.rgw.log++bucket.sync-target-hints.testbuck2 to cache LRU end 2023-01-24T17:34:00.470 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 10 chain_cache_entry: cache_locator= 2023-01-24T17:34:00.470 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:34:00.471 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:34:00.471 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-01-24T17:34:00.471 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.465+0000 7f37ead01680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :testbuck2[60550c17-6c7c-4ff3-b915-efe39561b124.4215.539]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-01-24T17:34:00.471 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.466+0000 7f37ead01680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-01-24T17:34:00.472 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-01-24T17:34:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-01-24T17:34:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-01-24T17:34:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-01-24T17:34:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":testbuck2[60550c17-6c7c-4ff3-b915-efe39561b124.4215.539])"; uploads.size()=0; is_truncated=0 2023-01-24T17:34:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:testbuck2[60550c17-6c7c-4ff3-b915-efe39561b124.4215.539]) 2023-01-24T17:34:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 20 lifecycle: get_system_obj_state: rctx=0x7ffec1ec0f88 obj=default.rgw.meta:root:testbuck2 state=0x55f30d6e5940 s->prefetch_data=0 2023-01-24T17:34:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : miss 2023-01-24T17:34:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-01-24T17:34:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 10 lifecycle: adding default.rgw.meta+root+testbuck2 to cache LRU end 2023-01-24T17:34:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.475 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:34:00.475 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 20 lifecycle: Read xattr: user.rgw.idtag 2023-01-24T17:34:00.475 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:34:00.475 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.468+0000 7f37ead01680 20 lifecycle: rados->read ofs=0 len=0 2023-01-24T17:34:00.475 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.469+0000 7f37ead01680 20 lifecycle: rados_obj.operate() r=0 bl.length=191 2023-01-24T17:34:00.475 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.469+0000 7f37ead01680 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x13 2023-01-24T17:34:00.476 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.469+0000 7f37ead01680 10 lifecycle: moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-01-24T17:34:00.476 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.469+0000 7f37ead01680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.471+0000 7f37ead01680 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 info.flags=0x17 2023-01-24T17:34:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.471+0000 7f37ead01680 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 to cache LRU end 2023-01-24T17:34:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.471+0000 7f37ead01680 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:34:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.471+0000 7f37ead01680 10 lifecycle: distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539, ofs0, ns] 2023-01-24T17:34:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.471+0000 7f37d77fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856069 cookie 94502392353104 notifier 4255 bl.length()=810 2023-01-24T17:34:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.471+0000 7f37d77fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 info.flags=0x17 2023-01-24T17:34:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.471+0000 7f37d77fe700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 to cache LRU end 2023-01-24T17:34:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.471+0000 7f37d77fe700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:34:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.471+0000 7f37ead01680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[60550c17-6c7c-4ff3-b915-efe39561b124.4215.539]): orig_sources= new_sources= 2023-01-24T17:34:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.471+0000 7f37ead01680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[60550c17-6c7c-4ff3-b915-efe39561b124.4215.539]): potential sources added=[] removed=[] 2023-01-24T17:34:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.471+0000 7f37ead01680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[60550c17-6c7c-4ff3-b915-efe39561b124.4215.539]): orig_dests= new_dests= 2023-01-24T17:34:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.471+0000 7f37ead01680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[60550c17-6c7c-4ff3-b915-efe39561b124.4215.539]): potential dests added=[] removed=[] 2023-01-24T17:34:00.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.481+0000 7f37ead01680 20 get_system_obj_state: rctx=0x7ffec1ec09e8 obj=default.rgw.meta:root:testbuck2 state=0x55f30d6e5940 s->prefetch_data=0 2023-01-24T17:34:00.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.481+0000 7f37ead01680 10 cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x16, cached=0x13) 2023-01-24T17:34:00.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.481+0000 7f37ead01680 10 cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-01-24T17:34:00.484 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.481+0000 7f37ead01680 10 moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-01-24T17:34:00.485 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.481+0000 7f37ead01680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.485 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.481+0000 7f37ead01680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:34:00.485 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.481+0000 7f37ead01680 10 cache get: name=default.rgw.meta+root+testbuck2 : hit (requested=0x11, cached=0x17) 2023-01-24T17:34:00.485 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.481+0000 7f37ead01680 10 removing default.rgw.meta+root+testbuck2 from cache 2023-01-24T17:34:00.485 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.481+0000 7f37ead01680 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:testbuck2, ofs0, ns] 2023-01-24T17:34:00.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.482+0000 7f37d6ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856070 cookie 94502392361712 notifier 4255 bl.length()=153 2023-01-24T17:34:00.486 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.483+0000 7f37ead01680 10 removing default.rgw.meta+root+.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 from cache 2023-01-24T17:34:00.487 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.483+0000 7f37ead01680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.testbuck2:60550c17-6c7c-4ff3-b915-efe39561b124.4215.539, ofs0, ns] 2023-01-24T17:34:00.487 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.484+0000 7f37d77fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856071 cookie 94502392353104 notifier 4255 bl.length()=212 2023-01-24T17:34:00.498 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.495+0000 7f37ead01680 10 removing default.rgw.log++pubsub..bucket.testbuck2/60550c17-6c7c-4ff3-b915-efe39561b124.4215.539 from cache 2023-01-24T17:34:00.498 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.495+0000 7f37ead01680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.testbuck2/60550c17-6c7c-4ff3-b915-efe39561b124.4215.539, ofs0, ns] 2023-01-24T17:34:00.498 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.495+0000 7f37d6ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856363 cookie 94502392349792 notifier 4255 bl.length()=209 2023-01-24T17:34:00.501 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.499+0000 7f37ead01680 10 removing key index: AMQVQYCNKWIFDXUYMOIN 2023-01-24T17:34:00.501 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.499+0000 7f37ead01680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.keys:AMQVQYCNKWIFDXUYMOIN, ofs0, ns] 2023-01-24T17:34:00.502 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.499+0000 7f37d77fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856364 cookie 94502392369680 notifier 4255 bl.length()=170 2023-01-24T17:34:00.503 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.500+0000 7f37ead01680 10 removing email index: foo.client.0+test@test.test 2023-01-24T17:34:00.503 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.500+0000 7f37ead01680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-01-24T17:34:00.503 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.500+0000 7f37d6ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856072 cookie 94502392368576 notifier 4255 bl.length()=178 2023-01-24T17:34:00.506 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.503+0000 7f37ead01680 10 removing user buckets index 2023-01-24T17:34:00.506 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.503+0000 7f37ead01680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2023-01-24T17:34:00.506 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.503+0000 7f37d77fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856073 cookie 94502392353104 notifier 4255 bl.length()=169 2023-01-24T17:34:00.508 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.505+0000 7f37ead01680 10 removing user index: foo.client.0 2023-01-24T17:34:00.508 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.505+0000 7f37ead01680 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2023-01-24T17:34:00.508 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.505+0000 7f37ead01680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-01-24T17:34:00.509 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.505+0000 7f37d6ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856365 cookie 94502392349792 notifier 4255 bl.length()=161 2023-01-24T17:34:00.510 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.508+0000 7f37ead01680 20 remove_watcher() i=0 2023-01-24T17:34:00.510 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.508+0000 7f37ead01680 2 removed watcher, disabling cache 2023-01-24T17:34:00.512 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.510+0000 7f37ead01680 20 remove_watcher() i=1 2023-01-24T17:34:00.514 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.512+0000 7f37ead01680 20 remove_watcher() i=2 2023-01-24T17:34:00.517 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.515+0000 7f37ead01680 20 remove_watcher() i=3 2023-01-24T17:34:00.518 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.515+0000 7f37ead01680 20 remove_watcher() i=4 2023-01-24T17:34:00.520 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.518+0000 7f37ead01680 20 remove_watcher() i=5 2023-01-24T17:34:00.523 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.520+0000 7f37ead01680 20 remove_watcher() i=6 2023-01-24T17:34:00.523 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.521+0000 7f37ead01680 20 remove_watcher() i=7 2023-01-24T17:34:00.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.522+0000 7f37af7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:34:00.529 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2023-01-24T17:34:00.608 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:34:00.608 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:34:00.613 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.611+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.615 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.612+0000 7f4213cd1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.615 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.612+0000 7f4213cd1680 20 realm 2023-01-24T17:34:00.615 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.612+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.616 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.612+0000 7f4213cd1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.616 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.612+0000 7f4213cd1680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:34:00.616 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.612+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.616 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.612+0000 7f4213cd1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.616 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.612+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.617 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.612+0000 7f4213cd1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.617 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.612+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.617 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.613+0000 7f4213cd1680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:34:00.618 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.613+0000 7f4213cd1680 20 searching for the correct realm 2023-01-24T17:34:00.623 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.620+0000 7f4213cd1680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:34:00.624 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.620+0000 7f4213cd1680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:34:00.624 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.620+0000 7f4213cd1680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:34:00.624 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.620+0000 7f4213cd1680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:34:00.624 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.620+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.624 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.620+0000 7f4213cd1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.625 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.620+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.625 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.620+0000 7f4213cd1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.625 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.620+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.625 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.620+0000 7f4213cd1680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:34:00.625 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.620+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.621+0000 7f4213cd1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.621+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.621+0000 7f4213cd1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.621+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.626 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.621+0000 7f4213cd1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.630 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.627+0000 7f4213cd1680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:34:00.630 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.627+0000 7f4213cd1680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:34:00.631 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.627+0000 7f4213cd1680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:34:00.631 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.627+0000 7f4213cd1680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:34:00.631 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.627+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.631 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.627+0000 7f4213cd1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.631 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.627+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.632 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:34:00.632 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.632 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.632 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.634 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:34:00.634 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.634 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.634 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 20 zone default found 2023-01-24T17:34:00.635 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 4 Realm: () 2023-01-24T17:34:00.635 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 4 ZoneGroup: default (ea8e6aae-de17-4da7-a2fe-dcef04325757) 2023-01-24T17:34:00.635 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 4 Zone: default (60550c17-6c7c-4ff3-b915-efe39561b124) 2023-01-24T17:34:00.635 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:34:00.635 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 20 zonegroup default 2023-01-24T17:34:00.636 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.636 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.636 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.628+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.636 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.629+0000 7f4213cd1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.636 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.629+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.637 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.629+0000 7f4213cd1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.637 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.629+0000 7f4213cd1680 20 started sync module instance, tier type = 2023-01-24T17:34:00.637 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.629+0000 7f4213cd1680 20 started zone id=60550c17-6c7c-4ff3-b915-efe39561b124 (name=default) with tier type = 2023-01-24T17:34:00.660 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.657+0000 7f4213cd1680 20 add_watcher() i=0 2023-01-24T17:34:00.660 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.657+0000 7f4213cd1680 20 add_watcher() i=1 2023-01-24T17:34:00.660 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.657+0000 7f4213cd1680 20 add_watcher() i=2 2023-01-24T17:34:00.661 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.657+0000 7f4213cd1680 20 add_watcher() i=3 2023-01-24T17:34:00.661 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.657+0000 7f4213cd1680 20 add_watcher() i=4 2023-01-24T17:34:00.661 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.657+0000 7f4213cd1680 20 add_watcher() i=5 2023-01-24T17:34:00.661 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.657+0000 7f4213cd1680 20 add_watcher() i=6 2023-01-24T17:34:00.662 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.660+0000 7f4213cd1680 20 add_watcher() i=7 2023-01-24T17:34:00.663 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.660+0000 7f4213cd1680 2 all 8 watchers are set, enabling cache 2023-01-24T17:34:00.665 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.662+0000 7f4153fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:34:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.663+0000 7f4213cd1680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:34:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.663+0000 7f4213cd1680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:34:00.718 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.714+0000 7f414affd700 20 reqs_thread_entry: start 2023-01-24T17:34:00.718 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.714+0000 7f4148ff9700 20 reqs_thread_entry: start 2023-01-24T17:34:00.718 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.714+0000 7f4213cd1680 20 init_complete bucket index max shards: 11 2023-01-24T17:34:00.718 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.714+0000 7f4213cd1680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:34:00.719 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.714+0000 7f4136ffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:34:31 2023 2023-01-24T17:34:00.719 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-01-24T17:34:00.719 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.714+0000 7f41367fc700 20 reqs_thread_entry: start 2023-01-24T17:34:00.719 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.714+0000 7f4213cd1680 20 get_system_obj_state: rctx=0x7ffc8413b1a8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x555763db5940 s->prefetch_data=0 2023-01-24T17:34:00.719 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.714+0000 7f4213cd1680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-01-24T17:34:00.720 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-01-24T17:34:00.720 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:34:00.720 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.720 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:34:00.720 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 20 Read xattr: user.rgw.idtag 2023-01-24T17:34:00.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:34:00.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 20 rados_obj.operate() r=0 bl.length=469 2023-01-24T17:34:00.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-01-24T17:34:00.721 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:34:00.722 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.722 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 10 removing key index: AJCMEGZGFSBXIBVXVBMR 2023-01-24T17:34:00.722 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.715+0000 7f4213cd1680 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.keys:AJCMEGZGFSBXIBVXVBMR, ofs0, ns] 2023-01-24T17:34:00.723 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.716+0000 7f4204dd0700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856074 cookie 93833827447312 notifier 4261 bl.length()=170 2023-01-24T17:34:00.723 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.717+0000 7f4213cd1680 10 removing email index: bar.client.0+test@test.test 2023-01-24T17:34:00.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.717+0000 7f4213cd1680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-01-24T17:34:00.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.717+0000 7f41fffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856075 cookie 93833827437488 notifier 4261 bl.length()=178 2023-01-24T17:34:00.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.718+0000 7f4213cd1680 10 removing user buckets index 2023-01-24T17:34:00.724 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.718+0000 7f4213cd1680 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2023-01-24T17:34:00.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.718+0000 7f4204dd0700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323857088 cookie 93833827441856 notifier 4261 bl.length()=169 2023-01-24T17:34:00.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.720+0000 7f4213cd1680 10 removing user index: bar.client.0 2023-01-24T17:34:00.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.720+0000 7f4213cd1680 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2023-01-24T17:34:00.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.720+0000 7f4213cd1680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-01-24T17:34:00.725 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.720+0000 7f41fffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856076 cookie 93833827437488 notifier 4261 bl.length()=161 2023-01-24T17:34:00.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.722+0000 7f4213cd1680 20 remove_watcher() i=0 2023-01-24T17:34:00.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.722+0000 7f4213cd1680 2 removed watcher, disabling cache 2023-01-24T17:34:00.726 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.723+0000 7f4213cd1680 20 remove_watcher() i=1 2023-01-24T17:34:00.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.724+0000 7f4213cd1680 20 remove_watcher() i=2 2023-01-24T17:34:00.727 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.725+0000 7f4213cd1680 20 remove_watcher() i=3 2023-01-24T17:34:00.728 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.725+0000 7f4213cd1680 20 remove_watcher() i=4 2023-01-24T17:34:00.729 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.726+0000 7f4213cd1680 20 remove_watcher() i=5 2023-01-24T17:34:00.730 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.727+0000 7f4213cd1680 20 remove_watcher() i=6 2023-01-24T17:34:00.730 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.728+0000 7f4213cd1680 20 remove_watcher() i=7 2023-01-24T17:34:00.731 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.729+0000 7f4153fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:34:00.736 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2023-01-24T17:34:00.814 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:34:00.815 INFO:teuthology.orchestra.run.smithi150.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:34:00.819 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.817+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.818+0000 7f87da7ba680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.823 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.818+0000 7f87da7ba680 20 realm 2023-01-24T17:34:00.823 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.818+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.823 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.818+0000 7f87da7ba680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.824 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.818+0000 7f87da7ba680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:34:00.824 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.818+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.824 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.818+0000 7f87da7ba680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.825 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.818+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.825 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.819+0000 7f87da7ba680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.825 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.819+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.826 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.819+0000 7f87da7ba680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:34:00.826 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.819+0000 7f87da7ba680 20 searching for the correct realm 2023-01-24T17:34:00.830 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.826+0000 7f87da7ba680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:34:00.830 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.826+0000 7f87da7ba680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:34:00.830 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.826+0000 7f87da7ba680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:34:00.831 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.826+0000 7f87da7ba680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:34:00.831 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.826+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.831 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.827+0000 7f87da7ba680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.831 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.827+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.831 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.827+0000 7f87da7ba680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.831 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.827+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.832 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.827+0000 7f87da7ba680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:34:00.976 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.827+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.977 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.827+0000 7f87da7ba680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.977 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.827+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.977 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.827+0000 7f87da7ba680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.827+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.987 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.828+0000 7f87da7ba680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.987 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.834+0000 7f87da7ba680 20 RGWRados::pool_iterate: got zonegroup_info.ea8e6aae-de17-4da7-a2fe-dcef04325757 2023-01-24T17:34:00.988 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.834+0000 7f87da7ba680 20 RGWRados::pool_iterate: got zone_info.60550c17-6c7c-4ff3-b915-efe39561b124 2023-01-24T17:34:00.988 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.834+0000 7f87da7ba680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:34:00.988 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.834+0000 7f87da7ba680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:34:00.988 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.834+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.988 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.834+0000 7f87da7ba680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.989 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.834+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.989 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.834+0000 7f87da7ba680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:34:00.989 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.834+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.989 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.834+0000 7f87da7ba680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:34:00.989 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.834+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.990 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:34:00.990 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.990 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.990 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 20 zone default found 2023-01-24T17:34:00.990 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 4 Realm: () 2023-01-24T17:34:00.991 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 4 ZoneGroup: default (ea8e6aae-de17-4da7-a2fe-dcef04325757) 2023-01-24T17:34:00.991 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 4 Zone: default (60550c17-6c7c-4ff3-b915-efe39561b124) 2023-01-24T17:34:00.991 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:34:00.991 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 20 zonegroup default 2023-01-24T17:34:00.991 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.992 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.992 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.992 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.992 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.835+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.992 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.836+0000 7f87da7ba680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:34:00.993 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.836+0000 7f87da7ba680 20 started sync module instance, tier type = 2023-01-24T17:34:00.993 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.836+0000 7f87da7ba680 20 started zone id=60550c17-6c7c-4ff3-b915-efe39561b124 (name=default) with tier type = 2023-01-24T17:34:00.993 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.846+0000 7f87da7ba680 20 add_watcher() i=0 2023-01-24T17:34:00.993 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.846+0000 7f87da7ba680 20 add_watcher() i=1 2023-01-24T17:34:00.993 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.846+0000 7f87da7ba680 20 add_watcher() i=2 2023-01-24T17:34:00.994 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.846+0000 7f87da7ba680 20 add_watcher() i=3 2023-01-24T17:34:00.994 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.846+0000 7f87da7ba680 20 add_watcher() i=4 2023-01-24T17:34:00.994 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.846+0000 7f87da7ba680 20 add_watcher() i=5 2023-01-24T17:34:00.994 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.848+0000 7f87da7ba680 20 add_watcher() i=6 2023-01-24T17:34:00.994 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.848+0000 7f87da7ba680 20 add_watcher() i=7 2023-01-24T17:34:00.995 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.848+0000 7f87da7ba680 2 all 8 watchers are set, enabling cache 2023-01-24T17:34:00.995 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.851+0000 7f879f7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:34:00.995 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.851+0000 7f87da7ba680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:34:00.995 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.851+0000 7f87da7ba680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:34:00.995 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.928+0000 7f870e7fc700 20 reqs_thread_entry: start 2023-01-24T17:34:00.996 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.928+0000 7f86fffff700 20 reqs_thread_entry: start 2023-01-24T17:34:00.996 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.928+0000 7f87da7ba680 20 init_complete bucket index max shards: 11 2023-01-24T17:34:00.996 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:34:00.996 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f86fe7fc700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:34:31 2023 2023-01-24T17:34:00.996 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-01-24T17:34:00.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f86fdffb700 20 reqs_thread_entry: start 2023-01-24T17:34:00.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 20 get_system_obj_state: rctx=0x7ffd2a2e5728 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55fb489c4940 s->prefetch_data=0 2023-01-24T17:34:00.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-01-24T17:34:00.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-01-24T17:34:00.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:34:00.998 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:00.998 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 10 updating xattr: name=user.rgw.user-policy bl.length()=118 2023-01-24T17:34:00.998 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:34:00.998 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 20 Read xattr: user.rgw.idtag 2023-01-24T17:34:00.998 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 20 Read xattr: user.rgw.user-policy 2023-01-24T17:34:00.999 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:34:00.999 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 20 rados->read ofs=0 len=0 2023-01-24T17:34:00.999 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 20 rados_obj.operate() r=0 bl.length=528 2023-01-24T17:34:00.999 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-01-24T17:34:00.999 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:34:01.000 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:34:01.000 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.929+0000 7f87da7ba680 10 updating xattr: name=user.rgw.user-policy bl.length()=118 2023-01-24T17:34:01.000 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.930+0000 7f87da7ba680 10 removing key index: DSPNUPXTHAQPETOJGKQM 2023-01-24T17:34:01.001 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.930+0000 7f87da7ba680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:DSPNUPXTHAQPETOJGKQM, ofs0, ns] 2023-01-24T17:34:01.001 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.930+0000 7f87bf7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856077 cookie 94537744956608 notifier 4267 bl.length()=170 2023-01-24T17:34:01.001 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.931+0000 7f87da7ba680 10 removing email index: testx$tenanteduser.client.0+test@test.test 2023-01-24T17:34:01.002 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.931+0000 7f87da7ba680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-01-24T17:34:01.002 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.932+0000 7f87beffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856366 cookie 94537744978848 notifier 4267 bl.length()=193 2023-01-24T17:34:01.003 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.933+0000 7f87da7ba680 10 removing user buckets index 2023-01-24T17:34:01.003 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.933+0000 7f87da7ba680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2023-01-24T17:34:01.003 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.933+0000 7f87beffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856078 cookie 94537744974400 notifier 4267 bl.length()=184 2023-01-24T17:34:01.003 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.935+0000 7f87da7ba680 10 removing user index: testx$tenanteduser.client.0 2023-01-24T17:34:01.004 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.935+0000 7f87da7ba680 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2023-01-24T17:34:01.004 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.935+0000 7f87da7ba680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-01-24T17:34:01.004 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.936+0000 7f87bf7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856367 cookie 94537744958944 notifier 4267 bl.length()=176 2023-01-24T17:34:01.005 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.940+0000 7f87da7ba680 20 remove_watcher() i=0 2023-01-24T17:34:01.005 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.940+0000 7f87da7ba680 2 removed watcher, disabling cache 2023-01-24T17:34:01.005 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.942+0000 7f87da7ba680 20 remove_watcher() i=1 2023-01-24T17:34:01.005 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.943+0000 7f87da7ba680 20 remove_watcher() i=2 2023-01-24T17:34:01.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.944+0000 7f87da7ba680 20 remove_watcher() i=3 2023-01-24T17:34:01.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.944+0000 7f87da7ba680 20 remove_watcher() i=4 2023-01-24T17:34:01.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.945+0000 7f87da7ba680 20 remove_watcher() i=5 2023-01-24T17:34:01.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.946+0000 7f87da7ba680 20 remove_watcher() i=6 2023-01-24T17:34:01.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.947+0000 7f87da7ba680 20 remove_watcher() i=7 2023-01-24T17:34:01.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-01-24T17:34:00.947+0000 7f879f7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:34:01.007 INFO:tasks.s3tests:Removing s3-tests... 2023-01-24T17:34:01.007 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/s3-tests 2023-01-24T17:34:01.137 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2023-01-24T17:34:01.152 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-01-24T17:34:01.164 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-01-24T17:34:01.165 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:34:01.166 INFO:tasks.rgw.client.0.smithi150.stdout:2023-01-24T17:34:01.162+0000 7f088f0be700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi150.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --rgw_crypt_sse_s3_vault_addr smithi150.front.sepia.ceph.com:8200 --rgw_crypt_sse_s3_vault_token_file /etc/ceph/vault-root-token --foreground (PID: 84958) UID: 0 2023-01-24T17:34:01.166 INFO:tasks.rgw.client.0.smithi150.stdout:2023-01-24T17:34:01.162+0000 7f08a0366740 -1 shutting down 2023-01-24T17:34:01.991 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2023-01-24T17:34:07.267 INFO:tasks.rgw.client.0:Stopped 2023-01-24T17:34:07.267 DEBUG:teuthology.orchestra.run.smithi150:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-01-24T17:34:07.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/vault-root-token 2023-01-24T17:34:07.365 DEBUG:teuthology.run_tasks:Unwinding manager vault 2023-01-24T17:34:07.374 INFO:tasks.vault:Stopping Vault instance 2023-01-24T17:34:07.374 DEBUG:tasks.vault.client.0:waiting for process to exit 2023-01-24T17:34:07.375 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:34:07.407 INFO:tasks.vault.client.0.smithi150.stdout:==> Vault shutdown triggered 2023-01-24T17:34:07.408 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:34:07.405Z [INFO] core: marked as sealed 2023-01-24T17:34:07.408 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:34:07.405Z [INFO] core: pre-seal teardown starting 2023-01-24T17:34:07.408 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:34:07.406Z [INFO] rollback: stopping rollback manager 2023-01-24T17:34:07.409 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:34:07.406Z [INFO] core: pre-seal teardown complete 2023-01-24T17:34:07.409 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:34:07.406Z [INFO] core: stopping cluster listeners 2023-01-24T17:34:07.409 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:34:07.406Z [INFO] core.cluster-listener: forwarding rpc listeners stopped 2023-01-24T17:34:07.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2023-01-24T17:34:07.788 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:34:07.787Z [INFO] core.cluster-listener: rpc listeners successfully shut down 2023-01-24T17:34:07.789 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:34:07.787Z [INFO] core: cluster listeners successfully shut down 2023-01-24T17:34:07.789 INFO:tasks.vault.client.0.smithi150.stderr:2023-01-24T17:34:07.787Z [INFO] core: vault is sealed 2023-01-24T17:34:07.790 INFO:tasks.vault.client.0:Stopped 2023-01-24T17:34:07.790 INFO:tasks.vault:Removing Vault... 2023-01-24T17:34:07.791 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/vault /home/ubuntu/cephtest/vault.zip 2023-01-24T17:34:07.865 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-01-24T17:34:07.874 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-01-24T17:34:07.875 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T17:34:08.217 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-01-24T17:34:08.217 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":35,"fsid":"f8f808ab-fc8c-410d-be49-462478484c6d","created":"2023-01-24T17:28:37.434400+0000","modified":"2023-01-24T17:30:45.472864+0000","last_up_change":"2023-01-24T17:28:45.042153+0000","last_in_change":"2023-01-24T17:28:38.394765+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T17:28:45.076462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T17:28:46.073389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-01-24T17:29:05.011239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-01-24T17:29:06.781817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-01-24T17:29:08.863183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-01-24T17:29:10.517462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-01-24T17:29:12.503095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-01-24T17:29:14.577223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2023-01-24T17:29:25.528687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2023-01-24T17:30:03.876187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"0b00efc2-6283-4c4f-b400-c4800e5b6679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":79855},{"type":"v1","addr":"172.21.15.150:6817","nonce":79855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":79855},{"type":"v1","addr":"172.21.15.150:6819","nonce":79855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":79855},{"type":"v1","addr":"172.21.15.150:6823","nonce":79855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":79855},{"type":"v1","addr":"172.21.15.150:6821","nonce":79855}]},"public_addr":"172.21.15.150:6817/79855","cluster_addr":"172.21.15.150:6819/79855","heartbeat_back_addr":"172.21.15.150:6823/79855","heartbeat_front_addr":"172.21.15.150:6821/79855","state":["exists","up"]},{"osd":1,"uuid":"cdbcc5c9-4004-429b-9c41-3098eb189f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":79845},{"type":"v1","addr":"172.21.15.150:6801","nonce":79845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":79845},{"type":"v1","addr":"172.21.15.150:6803","nonce":79845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":79845},{"type":"v1","addr":"172.21.15.150:6807","nonce":79845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":79845},{"type":"v1","addr":"172.21.15.150:6805","nonce":79845}]},"public_addr":"172.21.15.150:6801/79845","cluster_addr":"172.21.15.150:6803/79845","heartbeat_back_addr":"172.21.15.150:6807/79845","heartbeat_front_addr":"172.21.15.150:6805/79845","state":["exists","up"]},{"osd":2,"uuid":"9156c168-dc0b-4d3c-a01b-15a014386bbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":79846},{"type":"v1","addr":"172.21.15.150:6809","nonce":79846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":79846},{"type":"v1","addr":"172.21.15.150:6811","nonce":79846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":79846},{"type":"v1","addr":"172.21.15.150:6815","nonce":79846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":79846},{"type":"v1","addr":"172.21.15.150:6813","nonce":79846}]},"public_addr":"172.21.15.150:6809/79846","cluster_addr":"172.21.15.150:6811/79846","heartbeat_back_addr":"172.21.15.150:6815/79846","heartbeat_front_addr":"172.21.15.150:6813/79846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:28:42.771193+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:28:42.795644+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:28:42.763848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1","mappings":[{"from":0,"to":1}]},{"pgid":"3.30","mappings":[{"from":0,"to":1}]},{"pgid":"4.5","mappings":[{"from":1,"to":0}]},{"pgid":"4.8","mappings":[{"from":1,"to":0}]},{"pgid":"4.25","mappings":[{"from":1,"to":2}]},{"pgid":"4.2a","mappings":[{"from":1,"to":2}]},{"pgid":"4.33","mappings":[{"from":1,"to":0}]},{"pgid":"4.3a","mappings":[{"from":1,"to":0}]},{"pgid":"4.3d","mappings":[{"from":1,"to":2}]},{"pgid":"10.c","mappings":[{"from":2,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:34:08.231 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-01-24T17:34:08.548 INFO:teuthology.misc:Shutting down mds daemons... 2023-01-24T17:34:08.548 INFO:teuthology.misc:Shutting down osd daemons... 2023-01-24T17:34:08.549 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-01-24T17:34:08.549 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:34:08.610 INFO:tasks.ceph.osd.0:Stopped 2023-01-24T17:34:08.611 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-01-24T17:34:08.611 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:34:08.670 INFO:tasks.ceph.osd.1:Stopped 2023-01-24T17:34:08.670 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-01-24T17:34:08.671 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:34:08.719 INFO:tasks.ceph.osd.2:Stopped 2023-01-24T17:34:08.720 INFO:teuthology.misc:Shutting down mgr daemons... 2023-01-24T17:34:08.720 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-01-24T17:34:08.721 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:34:08.742 INFO:tasks.ceph.mgr.x:Stopped 2023-01-24T17:34:08.743 INFO:teuthology.misc:Shutting down mon daemons... 2023-01-24T17:34:08.743 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-01-24T17:34:08.743 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:34:08.793 INFO:tasks.ceph.mon.a:Stopped 2023-01-24T17:34:08.793 INFO:tasks.ceph:Checking cluster log for badness... 2023-01-24T17:34:08.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-01-24T17:34:08.859 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi150.front.sepia.ceph.com 2023-01-24T17:34:08.859 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-01-24T17:34:09.194 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi150.front.sepia.ceph.com 2023-01-24T17:34:09.195 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-01-24T17:34:09.297 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi150.front.sepia.ceph.com 2023-01-24T17:34:09.297 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-01-24T17:34:09.366 INFO:tasks.ceph:Archiving mon data... 2023-01-24T17:34:09.367 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135512/data/mon.a.tgz 2023-01-24T17:34:09.367 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2023-01-24T17:34:09.381 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.oQfvCesBlX 2023-01-24T17:34:09.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.oQfvCesBlX 2023-01-24T17:34:09.511 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.oQfvCesBlX 2023-01-24T17:34:09.592 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.oQfvCesBlX is 279KB 2023-01-24T17:34:09.685 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.oQfvCesBlX 2023-01-24T17:34:09.699 INFO:tasks.ceph:Cleaning ceph cluster... 2023-01-24T17:34:09.699 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-01-24T17:34:09.761 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-01-24T17:34:09.761 DEBUG:teuthology.orchestra.run.smithi150:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-01-24T17:34:09.826 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-01-24T17:34:09.831 INFO:tasks.ceph:Archiving crash dumps... 2023-01-24T17:34:09.833 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135512/remote/smithi150/crash 2023-01-24T17:34:09.834 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2023-01-24T17:34:09.899 INFO:tasks.ceph:Compressing logs... 2023-01-24T17:34:09.899 DEBUG:teuthology.orchestra.run.smithi150:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-01-24T17:34:37.272 INFO:tasks.ceph:Archiving logs... 2023-01-24T17:34:37.273 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135512/remote/smithi150/log 2023-01-24T17:34:37.274 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/log/ceph -- . 2023-01-24T17:34:42.723 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-01-24T17:34:42.731 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2023-01-24T17:34:42.732 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2023-01-24T17:34:42.808 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-01-24T17:34:42.818 INFO:teuthology.task.clock:Checking final clock skew... 2023-01-24T17:34:42.818 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T17:34:42.837 INFO:teuthology.orchestra.run.smithi150.stderr:bash: ntpq: command not found 2023-01-24T17:34:42.840 INFO:teuthology.orchestra.run.smithi150.stdout:210 Number of sources = 4 2023-01-24T17:34:42.840 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T17:34:42.840 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2023-01-24T17:34:42.840 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 35 +3999us[+3999us] +/- 101ms 2023-01-24T17:34:42.840 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 -1422us[-1419us] +/- 52ms 2023-01-24T17:34:42.841 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -838us[ -838us] +/- 74ms 2023-01-24T17:34:42.841 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T17:34:42.841 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-01-24T17:34:42.850 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-01-24T17:34:42.851 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-01-24T17:34:42.860 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-01-24T17:34:42.960 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T17:34:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T17:34:42.989 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-01-24T17:34:42.998 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-01-24T17:34:43.008 INFO:teuthology.task.internal:Duration was 1008.033663 seconds 2023-01-24T17:34:43.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-01-24T17:34:43.017 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-01-24T17:34:43.017 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-01-24T17:34:43.071 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T17:34:43.365 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-01-24T17:34:43.365 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2023-01-24T17:34:43.366 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-01-24T17:34:43.424 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-01-24T17:34:43.424 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-01-24T17:34:43.500 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-01-24T17:34:43.501 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-01-24T17:34:43.607 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-01-24T17:34:43.619 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-01-24T17:34:43.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-01-24T17:34:43.643 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-01-24T17:34:43.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-01-24T17:34:43.706 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2023-01-24T17:34:43.719 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2023-01-24T17:34:43.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T17:34:43.772 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-01-24T17:34:43.784 INFO:teuthology.task.internal:Transferring archived files... 2023-01-24T17:34:43.785 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135512/remote/smithi150 2023-01-24T17:34:43.785 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2023-01-24T17:34:43.858 INFO:teuthology.task.internal:Removing archive directory... 2023-01-24T17:34:43.859 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2023-01-24T17:34:43.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-01-24T17:34:43.921 INFO:teuthology.task.internal:Not uploading archives. 2023-01-24T17:34:43.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-01-24T17:34:43.930 INFO:teuthology.task.internal:Tidying up after the test... 2023-01-24T17:34:43.931 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-01-24T17:34:43.963 INFO:teuthology.orchestra.run.smithi150.stdout: 262168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 24 17:34 /home/ubuntu/cephtest 2023-01-24T17:34:43.964 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-01-24T17:34:43.986 INFO:teuthology.run:Summary data: description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} duration: 1008.0336625576019 flavor: default owner: scheduled_yuriw@teuthology success: true 2023-01-24T17:34:43.987 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T17:34:44.041 INFO:teuthology.run:pass