2023-10-05T19:48:46.720 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-05T19:48:46.721 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411774 --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} --verbose --name yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411774/orig.config.yaml 2023-10-05T19:48:46.752 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-05T19:48:46.826 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411774 branch: wip-yuri2-testing-2023-10-04-1333-quincy description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} email: yweinste@redhat.com first_in_suite: false job_id: '7411774' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri2-testing-2023-10-04-1333-quincy ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: vault rgw crypt sse s3 backend: vault rgw crypt sse s3 vault auth: token rgw crypt sse s3 vault prefix: /v1/transit/ rgw crypt sse s3 vault secret engine: transit rgw crypt vault auth: token rgw crypt vault prefix: /v1/transit/ rgw crypt vault secret engine: transit setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: default sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 rgw: client.0: use-vault-role: client.0 s3tests: with-sse-s3: true selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri2-testing-2023-10-04-1333-quincy sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 880 sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri2-testing-2023-10-04-1333-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 targets: smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8z/vefdvZUw41n3xNDmfIeVLkjdixVzTHNkrvMPaVIX3CjFH+aUP2pxblFIoZ876J04OLGO6n7SqTluvifork= tasks: - install: null - ceph: null - vault: client.0: engine: transit install_sha256: 7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 install_url: https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip prefix: /v1/transit/keys/ root_token: test_root_token secrets: - path: my-key-1 - path: my-key-2 - rgw: client.0: null - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 force-branch: ceph-quincy vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-05_14:30:58 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-05T19:48:46.826 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa; will attempt to use it 2023-10-05T19:48:46.827 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa/tasks 2023-10-05T19:48:46.827 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-05T19:48:46.829 INFO:teuthology.task.internal:Checking packages... 2023-10-05T19:48:46.851 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '487ffcb5ddb60d29c5688d1a457e812b1b697b90' 2023-10-05T19:48:46.851 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-05T19:48:46.851 INFO:teuthology.packaging:ref: None 2023-10-05T19:48:46.851 INFO:teuthology.packaging:tag: None 2023-10-05T19:48:46.851 INFO:teuthology.packaging:branch: wip-yuri2-testing-2023-10-04-1333-quincy 2023-10-05T19:48:46.851 INFO:teuthology.packaging:sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:48:46.852 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2023-10-04-1333-quincy 2023-10-05T19:48:47.119 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1278.g487ffcb5 2023-10-05T19:48:47.119 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-05T19:48:47.126 INFO:teuthology.task.internal:no buildpackages task found 2023-10-05T19:48:47.126 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-05T19:48:47.133 INFO:teuthology.task.internal:Saving configuration 2023-10-05T19:48:47.145 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-05T19:48:47.151 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T19:48:47.205 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411774', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:43:47.353685', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8z/vefdvZUw41n3xNDmfIeVLkjdixVzTHNkrvMPaVIX3CjFH+aUP2pxblFIoZ876J04OLGO6n7SqTluvifork='} 2023-10-05T19:48:47.205 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-05T19:48:47.211 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-05T19:48:47.211 INFO:teuthology.run_tasks:Running task console_log... 2023-10-05T19:48:47.248 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2d3c3e5ee0>, signals=[15]) 2023-10-05T19:48:47.248 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-05T19:48:47.254 INFO:teuthology.task.internal:Opening connections... 2023-10-05T19:48:47.255 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2023-10-05T19:48:47.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:48:47.326 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-05T19:48:47.332 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2023-10-05T19:48:47.398 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2023-10-05T19:48:47.398 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2023-10-05T19:48:47.458 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2023-10-05T19:48:47.459 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="8" 2023-10-05T19:48:47.459 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2023-10-05T19:48:47.459 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2023-10-05T19:48:47.459 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="8" 2023-10-05T19:48:47.459 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el8" 2023-10-05T19:48:47.459 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-05T19:48:47.459 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2023-10-05T19:48:47.459 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-05T19:48:47.459 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2023-10-05T19:48:47.459 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-05T19:48:47.459 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-05T19:48:47.460 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-05T19:48:47.460 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2023-10-05T19:48:47.479 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-05T19:48:47.488 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-05T19:48:47.493 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-05T19:48:47.493 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2023-10-05T19:48:47.517 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-05T19:48:47.522 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-05T19:48:47.522 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2023-10-05T19:48:47.586 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-05T19:48:47.617 INFO:teuthology.run_tasks:Running task kernel... 2023-10-05T19:48:47.630 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-05T19:48:47.630 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-05T19:48:47.631 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-05T19:48:47.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:48:47.649 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2023-10-05T19:48:47.708 INFO:teuthology.orchestra.run.smithi070.stdout:4.18.0-489.el8.x86_64 2023-10-05T19:48:47.708 INFO:teuthology.task.kernel:Running kernel on smithi070: 4.18.0-489.el8.x86_64 2023-10-05T19:48:47.709 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum install -y kernel 2023-10-05T19:58:17.313 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - AppStream 60 kB/s | 33 MB 09:27 2023-10-05T19:58:24.195 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - BaseOS 55 MB/s | 48 MB 00:00 2023-10-05T19:58:31.503 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - Extras 38 kB/s | 18 kB 00:00 2023-10-05T19:58:31.978 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 6.9 kB 00:00 2023-10-05T19:58:32.272 INFO:teuthology.orchestra.run.smithi070.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-10-05T19:58:34.466 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 7.8 MB/s | 16 MB 00:02 2023-10-05T19:58:38.247 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 379 kB/s | 24 kB 00:00 2023-10-05T19:58:41.729 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-05T19:58:41.729 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-05T19:58:41.993 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T19:58:41.994 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T19:58:41.994 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2023-10-05T19:58:41.994 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T19:58:41.994 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T19:58:41.994 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-10-05T19:58:41.994 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2023-10-05T19:58:41.994 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-10-05T19:58:41.994 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-10-05T19:58:41.995 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T19:58:41.995 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T19:58:41.995 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T19:58:41.995 INFO:teuthology.orchestra.run.smithi070.stdout:Install 3 Packages 2023-10-05T19:58:41.995 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T19:58:41.995 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 88 M 2023-10-05T19:58:41.995 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 96 M 2023-10-05T19:58:41.996 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T19:58:42.298 INFO:teuthology.orchestra.run.smithi070.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 39 MB/s | 10 MB 00:00 2023-10-05T19:58:42.698 INFO:teuthology.orchestra.run.smithi070.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 54 MB/s | 35 MB 00:00 2023-10-05T19:58:43.090 INFO:teuthology.orchestra.run.smithi070.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 40 MB/s | 43 MB 00:01 2023-10-05T19:58:43.090 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:58:43.090 INFO:teuthology.orchestra.run.smithi070.stdout:Total 80 MB/s | 88 MB 00:01 2023-10-05T19:58:43.761 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T19:58:44.171 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T19:58:44.172 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T19:58:45.397 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T19:58:45.426 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T19:58:48.184 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T19:58:48.689 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-10-05T19:58:52.333 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-10-05T19:58:52.738 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-10-05T19:58:56.111 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-10-05T19:58:56.451 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-10-05T19:59:18.216 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-10-05T19:59:18.240 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-10-05T19:59:18.567 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-10-05T19:59:18.567 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-10-05T19:59:18.567 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-10-05T19:59:19.309 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-10-05T19:59:19.309 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T19:59:19.309 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T19:59:19.309 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-10-05T19:59:19.310 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-10-05T19:59:19.310 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T19:59:19.310 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T19:59:19.445 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2023-10-05T19:59:19.691 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-4.18.0-514.el8.x86_64 2023-10-05T19:59:19.691 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-10-05T19:59:19.691 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-10-05T19:59:19.692 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-05T19:59:19.692 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-05T19:59:19.692 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-05T19:59:19.692 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2023-10-05T19:59:19.692 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2023-10-05T19:59:19.899 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-4.18.0-514.el8.x86_64 2023-10-05T19:59:19.899 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-10-05T19:59:19.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo rpm -qi grub2-tools 2023-10-05T19:59:19.933 INFO:teuthology.orchestra.run.smithi070.stdout:Name : grub2-tools 2023-10-05T19:59:19.934 INFO:teuthology.orchestra.run.smithi070.stdout:Epoch : 1 2023-10-05T19:59:19.934 INFO:teuthology.orchestra.run.smithi070.stdout:Version : 2.02 2023-10-05T19:59:19.934 INFO:teuthology.orchestra.run.smithi070.stdout:Release : 129.el8 2023-10-05T19:59:19.934 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: x86_64 2023-10-05T19:59:19.934 INFO:teuthology.orchestra.run.smithi070.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-05T19:59:19.934 INFO:teuthology.orchestra.run.smithi070.stdout:Group : System Environment/Base 2023-10-05T19:59:19.934 INFO:teuthology.orchestra.run.smithi070.stdout:Size : 9265586 2023-10-05T19:59:19.934 INFO:teuthology.orchestra.run.smithi070.stdout:License : GPLv3+ 2023-10-05T19:59:19.934 INFO:teuthology.orchestra.run.smithi070.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-05T19:59:19.934 INFO:teuthology.orchestra.run.smithi070.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-05T19:59:19.935 INFO:teuthology.orchestra.run.smithi070.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-05T19:59:19.935 INFO:teuthology.orchestra.run.smithi070.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-05T19:59:19.935 INFO:teuthology.orchestra.run.smithi070.stdout:Relocations : (not relocatable) 2023-10-05T19:59:19.935 INFO:teuthology.orchestra.run.smithi070.stdout:Packager : CentOS BuildSystem 2023-10-05T19:59:19.935 INFO:teuthology.orchestra.run.smithi070.stdout:Vendor : CentOS 2023-10-05T19:59:19.935 INFO:teuthology.orchestra.run.smithi070.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-05T19:59:19.935 INFO:teuthology.orchestra.run.smithi070.stdout:Summary : Support tools for GRUB. 2023-10-05T19:59:19.935 INFO:teuthology.orchestra.run.smithi070.stdout:Description : 2023-10-05T19:59:19.935 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T19:59:19.935 INFO:teuthology.orchestra.run.smithi070.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-05T19:59:19.935 INFO:teuthology.orchestra.run.smithi070.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-05T19:59:19.935 INFO:teuthology.orchestra.run.smithi070.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-05T19:59:19.936 INFO:teuthology.orchestra.run.smithi070.stdout:hardware devices. 2023-10-05T19:59:19.936 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T19:59:19.936 INFO:teuthology.orchestra.run.smithi070.stdout:This subpackage provides tools for support of all platforms. 2023-10-05T19:59:19.937 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-05T19:59:19.937 INFO:teuthology.task.kernel:Updating grub on smithi070 to boot 4.18.0-514.el8.x86_64 2023-10-05T19:59:19.937 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-05T19:59:21.656 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2023-10-05T19:59:23.447 INFO:teuthology.orchestra.run.smithi070.stderr:done 2023-10-05T19:59:23.449 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2023-10-05T19:59:23.471 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.i20H4WNaNp 2023-10-05T19:59:23.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.i20H4WNaNp 2023-10-05T19:59:23.545 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.i20H4WNaNp 2023-10-05T19:59:23.685 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.i20H4WNaNp is 6KB 2023-10-05T19:59:23.733 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.i20H4WNaNp 2023-10-05T19:59:23.750 DEBUG:teuthology.orchestra.run.smithi070:> sudo /bin/ls /boot/loader/entries || true 2023-10-05T19:59:23.828 INFO:teuthology.orchestra.run.smithi070.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-05T19:59:23.829 INFO:teuthology.orchestra.run.smithi070.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-05T19:59:23.829 INFO:teuthology.orchestra.run.smithi070.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-05T19:59:23.829 INFO:teuthology.orchestra.run.smithi070.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-10-05T19:59:23.830 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-10-05T19:59:23.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2023-10-05T19:59:54.018 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-05T19:59:54.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2023-10-05T19:59:54.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T20:00:12.410 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2023-10-05T20:00:21.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2023-10-05T20:00:21.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T20:01:21.475 DEBUG:teuthology.orchestra.remote:timed out 2023-10-05T20:01:33.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2023-10-05T20:01:33.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T20:01:33.752 DEBUG:teuthology.orchestra.run.smithi070:> true 2023-10-05T20:01:34.018 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2023-10-05T20:01:34.018 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-514.el8.x86_64"... 2023-10-05T20:01:34.019 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2023-10-05T20:01:34.038 INFO:teuthology.orchestra.run.smithi070.stdout:4.18.0-514.el8.x86_64 2023-10-05T20:01:34.039 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-10-05T20:01:34.039 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-05T20:01:34.039 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-05T20:01:35.040 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-05T20:01:35.041 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-05T20:01:35.121 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2023-10-05T20:01:35.137 DEBUG:teuthology.parallel:result is None 2023-10-05T20:01:35.137 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-05T20:01:35.144 INFO:teuthology.task.internal:Creating test directory... 2023-10-05T20:01:35.144 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-05T20:01:35.165 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-05T20:01:35.173 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-05T20:01:35.185 INFO:teuthology.task.internal:Creating archive directory... 2023-10-05T20:01:35.185 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-05T20:01:35.297 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-05T20:01:35.305 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-05T20:01:35.306 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-05T20:01:35.345 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T20:01:35.356 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T20:01:35.358 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-05T20:01:35.366 INFO:teuthology.task.internal:Configuring sudo... 2023-10-05T20:01:35.366 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-05T20:01:35.430 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-05T20:01:35.441 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-05T20:01:35.441 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-05T20:01:35.490 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-05T20:01:35.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-05T20:01:35.631 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-05T20:01:35.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-05T20:01:35.761 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T20:01:35.761 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-05T20:01:35.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2023-10-05T20:01:35.923 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-05T20:01:36.102 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-05T20:01:36.111 INFO:teuthology.task.internal:Starting timer... 2023-10-05T20:01:36.111 INFO:teuthology.run_tasks:Running task pcp... 2023-10-05T20:01:36.177 INFO:teuthology.run_tasks:Running task selinux... 2023-10-05T20:01:36.192 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-05T20:01:36.193 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2023-10-05T20:01:36.281 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2023-10-05T20:01:36.282 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-05T20:01:36.282 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2023-10-05T20:01:36.330 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2023-10-05T20:01:36.331 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi070.front.sepia.ceph.com': 'permissive'} 2023-10-05T20:01:36.331 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-05T20:01:36.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T20:01:36.358 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-05T20:01:36.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2023-10-05T20:01:36.427 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-05T20:01:36.437 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-05T20:01:36.437 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-05T20:01:36.452 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-05T20:01:36.453 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi070.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-05T20:18:21.308 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi070.front.sepia.ceph.com')] 2023-10-05T20:18:21.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2023-10-05T20:18:21.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T20:18:21.380 DEBUG:teuthology.orchestra.run.smithi070:> true 2023-10-05T20:18:21.453 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2023-10-05T20:18:21.453 INFO:teuthology.run_tasks:Running task clock... 2023-10-05T20:18:21.470 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-05T20:18:21.470 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-05T20:18:21.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T20:18:21.524 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-05T20:18:21.538 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-05T20:18:21.602 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2023-10-05T20:18:21.613 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2023-10-05T20:18:21.627 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-05T20:18:21.641 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-05T20:18:21.712 INFO:teuthology.orchestra.run.smithi070.stderr:bash: ntpq: command not found 2023-10-05T20:18:21.715 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-05T20:18:21.715 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2023-10-05T20:18:21.715 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T20:18:21.715 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T20:18:21.715 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T20:18:21.715 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T20:18:21.716 INFO:teuthology.run_tasks:Running task install... 2023-10-05T20:18:21.727 DEBUG:teuthology.task.install:project ceph 2023-10-05T20:18:21.727 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}} 2023-10-05T20:18:21.727 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'} 2023-10-05T20:18:21.727 INFO:teuthology.task.install:Using flavor: default 2023-10-05T20:18:21.732 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-05T20:18:21.733 INFO:teuthology.task.install:extra packages: [] 2023-10-05T20:18:21.733 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'tag': None, 'wait_for_package': False} 2023-10-05T20:18:21.733 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T20:18:22.005 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/ 2023-10-05T20:18:22.005 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1278.g487ffcb5 2023-10-05T20:18:22.173 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-05T20:18:22.201 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T20:18:22.202 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-05T20:18:22.231 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-05T20:18:22.232 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/487ffcb5ddb60d29c5688d1a457e812b1b697b90/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-05T20:18:22.302 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-05T20:18:22.611 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-05T20:18:22.679 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2023-10-05T20:18:22.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2023-10-05T20:18:22.951 INFO:teuthology.orchestra.run.smithi070.stdout:76 files removed 2023-10-05T20:18:22.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw 2023-10-05T20:18:23.747 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 217 kB/s | 78 kB 00:00 2023-10-05T20:18:24.115 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 28 kB/s | 9.2 kB 00:00 2023-10-05T20:18:24.374 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 6.3 kB/s | 1.4 kB 00:00 2023-10-05T20:27:16.098 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - AppStream 64 kB/s | 33 MB 08:51 2023-10-05T20:27:23.056 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - BaseOS 54 MB/s | 48 MB 00:00 2023-10-05T20:27:30.372 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - Extras 39 kB/s | 18 kB 00:00 2023-10-05T20:27:30.698 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - Extras common packages 46 kB/s | 6.9 kB 00:00 2023-10-05T20:27:31.524 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 8 - PowerTools 9.6 MB/s | 6.5 MB 00:00 2023-10-05T20:27:32.942 INFO:teuthology.orchestra.run.smithi070.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-10-05T20:27:35.017 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 8.3 MB/s | 16 MB 00:01 2023-10-05T20:27:38.942 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 393 kB/s | 24 kB 00:00 2023-10-05T20:27:43.397 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T20:27:43.400 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:27:43.400 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-10-05T20:27:43.400 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:27:43.400 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T20:27:43.400 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 17 M 2023-10-05T20:27:43.400 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2023-10-05T20:27:43.400 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.5 M 2023-10-05T20:27:43.400 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.6 M 2023-10-05T20:27:43.400 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 5.2 M 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 23 M 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 24 k 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 797 k 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 506 k 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 5.6 M 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-05T20:27:43.401 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 45 k 2023-10-05T20:27:43.402 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 115 k 2023-10-05T20:27:43.402 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 191 k 2023-10-05T20:27:43.402 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-05T20:27:43.402 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 370 k 2023-10-05T20:27:43.402 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 359 k 2023-10-05T20:27:43.402 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 106 k 2023-10-05T20:27:43.402 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-05T20:27:43.402 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:27:43.402 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T20:27:43.402 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:27:43.402 INFO:teuthology.orchestra.run.smithi070.stdout:Install 21 Packages 2023-10-05T20:27:43.403 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 2 Packages 2023-10-05T20:27:43.403 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:27:43.403 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 63 M 2023-10-05T20:27:43.403 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T20:27:44.385 INFO:teuthology.orchestra.run.smithi070.stdout:(1/23): ceph-base-17.2.6-1278.g487ffcb5.el8.x86 8.4 MB/s | 5.2 MB 00:00 2023-10-05T20:27:44.452 INFO:teuthology.orchestra.run.smithi070.stdout:(2/23): ceph-selinux-17.2.6-1278.g487ffcb5.el8. 366 kB/s | 24 kB 00:00 2023-10-05T20:27:44.552 INFO:teuthology.orchestra.run.smithi070.stdout:(3/23): libcephfs2-17.2.6-1278.g487ffcb5.el8.x8 7.8 MB/s | 797 kB 00:00 2023-10-05T20:27:44.653 INFO:teuthology.orchestra.run.smithi070.stdout:(4/23): libradosstriper1-17.2.6-1278.g487ffcb5. 4.9 MB/s | 506 kB 00:00 2023-10-05T20:27:45.128 INFO:teuthology.orchestra.run.smithi070.stdout:(5/23): librgw2-17.2.6-1278.g487ffcb5.el8.x86_6 12 MB/s | 5.6 MB 00:00 2023-10-05T20:27:45.186 INFO:teuthology.orchestra.run.smithi070.stdout:(6/23): python3-ceph-argparse-17.2.6-1278.g487f 769 kB/s | 45 kB 00:00 2023-10-05T20:27:45.253 INFO:teuthology.orchestra.run.smithi070.stdout:(7/23): python3-ceph-common-17.2.6-1278.g487ffc 1.7 MB/s | 115 kB 00:00 2023-10-05T20:27:45.512 INFO:teuthology.orchestra.run.smithi070.stdout:(8/23): ceph-common-17.2.6-1278.g487ffcb5.el8.x 13 MB/s | 23 MB 00:01 2023-10-05T20:27:45.529 INFO:teuthology.orchestra.run.smithi070.stdout:(9/23): python3-cephfs-17.2.6-1278.g487ffcb5.el 692 kB/s | 191 kB 00:00 2023-10-05T20:27:45.563 INFO:teuthology.orchestra.run.smithi070.stdout:(10/23): python3-rados-17.2.6-1278.g487ffcb5.el 7.2 MB/s | 370 kB 00:00 2023-10-05T20:27:45.588 INFO:teuthology.orchestra.run.smithi070.stdout:(11/23): python3-rbd-17.2.6-1278.g487ffcb5.el8. 6.0 MB/s | 359 kB 00:00 2023-10-05T20:27:45.673 INFO:teuthology.orchestra.run.smithi070.stdout:(12/23): python3-rgw-17.2.6-1278.g487ffcb5.el8. 968 kB/s | 106 kB 00:00 2023-10-05T20:27:48.089 INFO:teuthology.orchestra.run.smithi070.stdout:(13/23): ceph-radosgw-17.2.6-1278.g487ffcb5.el8 3.9 MB/s | 17 MB 00:04 2023-10-05T20:31:53.516 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-prettytable-0.7.2-14.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-prettytable-0.7.2-14.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:31:53.679 INFO:teuthology.orchestra.run.smithi070.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 183 B/s | 44 kB 04:05 2023-10-05T20:31:53.780 INFO:teuthology.orchestra.run.smithi070.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 467 kB/s | 47 kB 00:00 2023-10-05T20:31:54.480 INFO:teuthology.orchestra.run.smithi070.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 437 kB/s | 306 kB 00:00 2023-10-05T20:31:54.522 INFO:teuthology.orchestra.run.smithi070.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2023-10-05T20:31:54.564 INFO:teuthology.orchestra.run.smithi070.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2023-10-05T20:31:54.914 INFO:teuthology.orchestra.run.smithi070.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 5.0 MB/s | 1.7 MB 00:00 2023-10-05T20:31:55.290 INFO:teuthology.orchestra.run.smithi070.stdout:(20/23): librados2-17.2.6-1278.g487ffcb5.el8.x8 9.2 MB/s | 3.5 MB 00:00 2023-10-05T20:31:55.530 INFO:teuthology.orchestra.run.smithi070.stdout:(21/23): librbd1-17.2.6-1278.g487ffcb5.el8.x86_ 15 MB/s | 3.6 MB 00:00 2023-10-05T20:33:01.607 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] librdkafka-0.11.4-3.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/librdkafka-0.11.4-3.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:33:01.607 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] lttng-ust-2.8.1-11.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/lttng-ust-2.8.1-11.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:33:02.219 INFO:teuthology.orchestra.run.smithi070.stdout:(22/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 839 B/s | 259 kB 05:16 2023-10-05T20:33:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:(23/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.1 kB/s | 354 kB 05:16 2023-10-05T20:33:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:33:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:Total 203 kB/s | 63 MB 05:18 2023-10-05T20:33:02.282 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T20:33:02.328 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T20:33:02.328 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T20:33:02.726 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T20:33:02.729 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T20:33:03.868 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T20:33:04.128 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-10-05T20:33:04.344 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-10-05T20:33:04.710 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-10-05T20:33:04.911 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/25 2023-10-05T20:33:05.017 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/25 2023-10-05T20:33:05.327 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 4/25 2023-10-05T20:33:05.544 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/25 2023-10-05T20:33:05.942 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/25 2023-10-05T20:33:06.177 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/25 2023-10-05T20:33:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/25 2023-10-05T20:33:06.606 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-10-05T20:33:06.784 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-10-05T20:33:06.977 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-05T20:33:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-05T20:33:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 10/25 2023-10-05T20:33:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 10/25 2023-10-05T20:33:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:17.2.6-1278.g487ffcb5.el8. 11/25 2023-10-05T20:33:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 12/25 2023-10-05T20:33:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 13/25 2023-10-05T20:33:08.672 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 14/25 2023-10-05T20:33:08.810 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_6 15/25 2023-10-05T20:33:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_6 15/25 2023-10-05T20:33:09.152 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-10-05T20:33:09.368 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-10-05T20:33:09.537 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-05T20:33:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:17.2.6-1278.g487ffcb5.el8.x8 19/25 2023-10-05T20:33:12.513 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 20/25 2023-10-05T20:33:12.759 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 20/25 2023-10-05T20:33:13.335 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 20/25 2023-10-05T20:33:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 21/25 2023-10-05T20:33:13.729 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 21/25 2023-10-05T20:33:13.909 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 22/25 2023-10-05T20:33:32.276 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 22/25 2023-10-05T20:33:32.277 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2023-10-05T20:33:32.277 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2023-10-05T20:33:32.277 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /dev 2023-10-05T20:33:32.277 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /run 2023-10-05T20:33:32.277 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2023-10-05T20:33:32.277 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2023-10-05T20:33:32.277 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2023-10-05T20:33:32.277 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2023-10-05T20:33:32.277 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /dev 2023-10-05T20:33:32.277 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:33:33.992 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 23/25 2023-10-05T20:33:34.065 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 23/25 2023-10-05T20:33:34.065 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-05T20:33:34.324 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-05T20:33:34.324 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-05T20:33:35.056 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-05T20:33:35.057 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/25 2023-10-05T20:33:35.057 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/25 2023-10-05T20:33:35.057 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/25 2023-10-05T20:33:35.057 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 4/25 2023-10-05T20:33:35.057 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/25 2023-10-05T20:33:35.057 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_6 6/25 2023-10-05T20:33:35.057 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 7/25 2023-10-05T20:33:35.057 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1278.g487ffcb5.el8. 8/25 2023-10-05T20:33:35.057 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:17.2.6-1278.g487ffcb5.el8.x8 9/25 2023-10-05T20:33:35.058 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 10/25 2023-10-05T20:33:35.058 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 11/25 2023-10-05T20:33:35.058 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 12/25 2023-10-05T20:33:35.058 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 13/25 2023-10-05T20:33:35.058 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-10-05T20:33:35.058 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-10-05T20:33:35.058 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-05T20:33:35.059 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-10-05T20:33:35.059 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-05T20:33:35.059 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-10-05T20:33:35.059 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-10-05T20:33:35.059 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-10-05T20:33:35.059 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 22/25 2023-10-05T20:33:35.059 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-05T20:33:35.059 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 24/25 2023-10-05T20:33:36.120 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-05T20:33:36.120 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:33:36.120 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2023-10-05T20:33:36.121 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.121 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.121 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T20:33:36.121 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.121 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.121 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.121 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.121 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-05T20:33:36.121 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.121 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-05T20:33:36.122 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-05T20:33:36.122 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.122 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-05T20:33:36.122 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.122 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-05T20:33:36.122 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-05T20:33:36.122 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.122 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.122 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.122 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-05T20:33:36.123 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.123 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.123 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:33:36.123 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-05T20:33:36.123 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:33:36.123 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T20:33:36.458 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-test 2023-10-05T20:33:37.306 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:05:59 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T20:33:38.888 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T20:33:38.889 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:33:38.889 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-10-05T20:33:38.889 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:33:38.889 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T20:33:38.889 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 52 M 2023-10-05T20:33:38.889 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2023-10-05T20:33:38.889 INFO:teuthology.orchestra.run.smithi070.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-05T20:33:38.889 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 174 k 2023-10-05T20:33:38.889 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-05T20:33:38.889 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-05T20:33:38.890 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-05T20:33:38.890 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:33:38.890 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T20:33:38.890 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:33:38.890 INFO:teuthology.orchestra.run.smithi070.stdout:Install 6 Packages 2023-10-05T20:33:38.890 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:33:38.891 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 53 M 2023-10-05T20:33:38.891 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 234 M 2023-10-05T20:33:38.891 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T20:33:39.673 INFO:teuthology.orchestra.run.smithi070.stdout:(1/6): libcephsqlite-17.2.6-1278.g487ffcb5.el8. 235 kB/s | 174 kB 00:00 2023-10-05T20:33:43.487 INFO:teuthology.orchestra.run.smithi070.stdout:(2/6): ceph-test-17.2.6-1278.g487ffcb5.el8.x86_ 11 MB/s | 52 MB 00:04 2023-10-05T20:38:02.100 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] oniguruma-6.8.2-2.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/oniguruma-6.8.2-2.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:38:02.543 INFO:teuthology.orchestra.run.smithi070.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 730 B/s | 187 kB 04:22 2023-10-05T20:38:39.659 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] jq-1.6-7.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/jq-1.6-7.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:38:40.301 INFO:teuthology.orchestra.run.smithi070.stdout:(4/6): jq-1.6-7.el8.x86_64.rpm 688 B/s | 202 kB 05:01 2023-10-05T20:38:43.662 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] socat-1.7.4.1-1.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-1.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:38:44.342 INFO:teuthology.orchestra.run.smithi070.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.1 kB/s | 323 kB 05:00 2023-10-05T20:42:11.935 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] xmlstarlet-1.6.1-20.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:42:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:(6/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 289 B/s | 71 kB 04:09 2023-10-05T20:42:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:42:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:Total 105 kB/s | 53 MB 08:33 2023-10-05T20:42:12.462 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T20:42:12.483 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T20:42:12.483 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T20:42:12.794 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T20:42:12.797 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T20:42:13.276 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T20:42:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-10-05T20:42:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-10-05T20:42:13.837 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-05T20:42:13.918 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-05T20:42:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-10-05T20:42:14.262 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/6 2023-10-05T20:42:19.310 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/6 2023-10-05T20:42:19.509 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/6 2023-10-05T20:42:19.792 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/6 2023-10-05T20:42:19.792 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/6 2023-10-05T20:42:19.792 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/6 2023-10-05T20:42:19.792 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-10-05T20:42:19.792 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-10-05T20:42:19.792 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-10-05T20:42:20.439 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-10-05T20:42:20.439 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:42:20.439 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T20:42:20.440 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:42:20.440 INFO:teuthology.orchestra.run.smithi070.stdout: jq-1.6-7.el8.x86_64 2023-10-05T20:42:20.440 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:42:20.440 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-05T20:42:20.440 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-05T20:42:20.440 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-05T20:42:20.440 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:42:20.440 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T20:42:20.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph 2023-10-05T20:42:21.505 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:14:43 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T20:42:23.099 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T20:42:23.102 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:42:23.102 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-10-05T20:42:23.102 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:42:23.102 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T20:42:23.103 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 6.4 k 2023-10-05T20:42:23.103 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2023-10-05T20:42:23.103 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 2.3 M 2023-10-05T20:42:23.103 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 1.6 M 2023-10-05T20:42:23.103 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 238 k 2023-10-05T20:42:23.103 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.9 M 2023-10-05T20:42:23.103 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 17 M 2023-10-05T20:42:23.103 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-05T20:42:23.103 INFO:teuthology.orchestra.run.smithi070.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-05T20:42:23.103 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-05T20:42:23.103 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-05T20:42:23.103 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-05T20:42:23.104 INFO:teuthology.orchestra.run.smithi070.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-05T20:42:23.105 INFO:teuthology.orchestra.run.smithi070.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-05T20:42:23.105 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-05T20:42:23.105 INFO:teuthology.orchestra.run.smithi070.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-05T20:42:23.105 INFO:teuthology.orchestra.run.smithi070.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-05T20:42:23.105 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-05T20:42:23.105 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-05T20:42:23.105 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-05T20:42:23.105 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-05T20:42:23.105 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:42:23.105 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T20:42:23.105 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:42:23.105 INFO:teuthology.orchestra.run.smithi070.stdout:Install 32 Packages 2023-10-05T20:42:23.106 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:42:23.106 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 28 M 2023-10-05T20:42:23.106 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 110 M 2023-10-05T20:42:23.106 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T20:42:23.464 INFO:teuthology.orchestra.run.smithi070.stdout:(1/32): ceph-17.2.6-1278.g487ffcb5.el8.x86_64.r 27 kB/s | 6.4 kB 00:00 2023-10-05T20:42:23.589 INFO:teuthology.orchestra.run.smithi070.stdout:(2/32): ceph-mgr-17.2.6-1278.g487ffcb5.el8.x86_ 4.3 MB/s | 1.6 MB 00:00 2023-10-05T20:42:24.090 INFO:teuthology.orchestra.run.smithi070.stdout:(3/32): ceph-mds-17.2.6-1278.g487ffcb5.el8.x86_ 2.7 MB/s | 2.3 MB 00:00 2023-10-05T20:42:24.173 INFO:teuthology.orchestra.run.smithi070.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1278.g487f 2.8 MB/s | 238 kB 00:00 2023-10-05T20:42:24.490 INFO:teuthology.orchestra.run.smithi070.stdout:(5/32): ceph-osd-17.2.6-1278.g487ffcb5.el8.x86_ 19 MB/s | 17 MB 00:00 2023-10-05T20:42:24.549 INFO:teuthology.orchestra.run.smithi070.stdout:(6/32): ceph-mon-17.2.6-1278.g487ffcb5.el8.x86_ 3.6 MB/s | 3.9 MB 00:01 2023-10-05T20:46:29.982 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-markupsafe-0.23-19.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-markupsafe-0.23-19.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:46:30.187 INFO:teuthology.orchestra.run.smithi070.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 161 B/s | 39 kB 04:05 2023-10-05T20:46:34.078 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-mako-1.0.6-14.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-mako-1.0.6-14.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:46:34.295 INFO:teuthology.orchestra.run.smithi070.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 642 B/s | 157 kB 04:10 2023-10-05T20:46:50.462 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-pyOpenSSL-19.0.0-1.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pyOpenSSL-19.0.0-1.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:46:50.700 INFO:teuthology.orchestra.run.smithi070.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 398 B/s | 103 kB 04:26 2023-10-05T20:46:50.800 INFO:teuthology.orchestra.run.smithi070.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.4 MB/s | 249 kB 00:00 2023-10-05T20:46:50.850 INFO:teuthology.orchestra.run.smithi070.stdout:(11/32): python3-cryptography-3.2.1-6.el8.x86_6 13 MB/s | 641 kB 00:00 2023-10-05T20:46:50.884 INFO:teuthology.orchestra.run.smithi070.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 3.2 MB/s | 109 kB 00:00 2023-10-05T20:46:51.176 INFO:teuthology.orchestra.run.smithi070.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 149 kB/s | 44 kB 00:00 2023-10-05T20:46:51.384 INFO:teuthology.orchestra.run.smithi070.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 889 kB/s | 185 kB 00:00 2023-10-05T20:46:51.501 INFO:teuthology.orchestra.run.smithi070.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.4 MB/s | 173 kB 00:00 2023-10-05T20:46:51.651 INFO:teuthology.orchestra.run.smithi070.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.5 MB/s | 384 kB 00:00 2023-10-05T20:46:51.752 INFO:teuthology.orchestra.run.smithi070.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2023-10-05T20:46:51.836 INFO:teuthology.orchestra.run.smithi070.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 211 kB/s | 18 kB 00:00 2023-10-05T20:46:51.919 INFO:teuthology.orchestra.run.smithi070.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 585 kB/s | 49 kB 00:00 2023-10-05T20:46:52.003 INFO:teuthology.orchestra.run.smithi070.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 704 kB/s | 59 kB 00:00 2023-10-05T20:46:52.103 INFO:teuthology.orchestra.run.smithi070.stdout:(21/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2023-10-05T20:46:52.178 INFO:teuthology.orchestra.run.smithi070.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 208 kB/s | 16 kB 00:00 2023-10-05T20:46:52.262 INFO:teuthology.orchestra.run.smithi070.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 225 kB/s | 19 kB 00:00 2023-10-05T20:46:52.337 INFO:teuthology.orchestra.run.smithi070.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 319 kB/s | 24 kB 00:00 2023-10-05T20:46:52.413 INFO:teuthology.orchestra.run.smithi070.stdout:(25/32): python3-tempora-1.14.1-5.el8.noarch.rp 390 kB/s | 29 kB 00:00 2023-10-05T20:46:52.488 INFO:teuthology.orchestra.run.smithi070.stdout:(26/32): python3-trustme-0.6.0-4.el8.noarch.rpm 366 kB/s | 27 kB 00:00 2023-10-05T20:46:52.580 INFO:teuthology.orchestra.run.smithi070.stdout:(27/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2023-10-05T20:46:52.672 INFO:teuthology.orchestra.run.smithi070.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2023-10-05T20:46:52.755 INFO:teuthology.orchestra.run.smithi070.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2023-10-05T20:46:52.848 INFO:teuthology.orchestra.run.smithi070.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 249 kB/s | 23 kB 00:00 2023-10-05T20:51:30.794 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-pytz-2017.2-11.el8.noarch.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pytz-2017.2-11.el8.noarch.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:51:31.511 INFO:teuthology.orchestra.run.smithi070.stdout:(31/32): python3-pytz-2017.2-11.el8.noarch.rpm 194 B/s | 57 kB 05:01 2023-10-05T20:51:35.296 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-werkzeug-0.12.2-4.el8.noarch.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-werkzeug-0.12.2-4.el8.noarch.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:51:35.894 INFO:teuthology.orchestra.run.smithi070.stdout:(32/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.5 kB/s | 457 kB 05:01 2023-10-05T20:51:35.894 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:51:35.895 INFO:teuthology.orchestra.run.smithi070.stdout:Total 53 kB/s | 28 MB 09:12 2023-10-05T20:51:35.972 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T20:51:36.009 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T20:51:36.009 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T20:51:36.326 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T20:51:36.329 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T20:51:37.406 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T20:51:37.599 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-05T20:51:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-05T20:51:37.994 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-10-05T20:51:38.200 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-10-05T20:51:38.372 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-10-05T20:51:38.569 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-10-05T20:51:38.729 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-10-05T20:51:38.948 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-10-05T20:51:39.148 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-10-05T20:51:39.360 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-10-05T20:51:39.558 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-10-05T20:51:39.773 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-10-05T20:51:40.134 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-10-05T20:51:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-10-05T20:51:40.564 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-10-05T20:51:40.794 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-10-05T20:51:41.011 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-05T20:51:41.314 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-10-05T20:51:41.482 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-10-05T20:51:41.695 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-10-05T20:51:41.873 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-05T20:51:42.129 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-05T20:51:42.337 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-05T20:51:42.556 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-10-05T20:51:42.858 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-10-05T20:51:43.140 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-05T20:51:43.485 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1278.g487ffcb5.el8. 27/32 2023-10-05T20:51:43.663 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 28/32 2023-10-05T20:51:45.140 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 28/32 2023-10-05T20:51:45.288 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 29/32 2023-10-05T20:51:45.680 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 29/32 2023-10-05T20:51:45.842 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 30/32 2023-10-05T20:51:46.058 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 30/32 2023-10-05T20:51:46.209 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 31/32 2023-10-05T20:51:46.226 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 31/32 2023-10-05T20:51:46.333 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 32/32 2023-10-05T20:51:46.744 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 32/32 2023-10-05T20:51:46.744 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/32 2023-10-05T20:51:46.744 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/32 2023-10-05T20:51:46.744 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/32 2023-10-05T20:51:46.744 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 4/32 2023-10-05T20:51:46.744 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/32 2023-10-05T20:51:46.744 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1278.g487ffcb5.el8. 6/32 2023-10-05T20:51:46.744 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-10-05T20:51:46.745 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-10-05T20:51:46.746 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-10-05T20:51:46.746 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-10-05T20:51:46.746 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-10-05T20:51:46.746 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-10-05T20:51:46.746 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-10-05T20:51:46.746 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-10-05T20:51:46.746 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-10-05T20:51:46.747 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-10-05T20:51:46.747 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-10-05T20:51:46.747 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-10-05T20:51:46.747 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-10-05T20:51:47.813 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-10-05T20:51:47.813 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:51:47.813 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T20:51:47.813 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:51:47.814 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:51:47.814 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:51:47.814 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:51:47.814 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:51:47.814 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:51:47.814 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-05T20:51:47.814 INFO:teuthology.orchestra.run.smithi070.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-05T20:51:47.814 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-05T20:51:47.814 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-05T20:51:47.814 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-05T20:51:47.814 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-05T20:51:47.814 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-05T20:51:47.815 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-05T20:51:47.815 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-05T20:51:47.815 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-05T20:51:47.815 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-05T20:51:47.815 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-05T20:51:47.815 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-05T20:51:47.815 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-2.6-1.el8.noarch 2023-10-05T20:51:47.815 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-05T20:51:47.815 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-05T20:51:47.815 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-05T20:51:47.815 INFO:teuthology.orchestra.run.smithi070.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-05T20:51:47.815 INFO:teuthology.orchestra.run.smithi070.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-05T20:51:47.816 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-05T20:51:47.816 INFO:teuthology.orchestra.run.smithi070.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-05T20:51:47.816 INFO:teuthology.orchestra.run.smithi070.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-05T20:51:47.816 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-05T20:51:47.816 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-05T20:51:47.816 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-05T20:51:47.816 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-05T20:51:47.816 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:51:47.816 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T20:51:48.091 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-base 2023-10-05T20:51:48.946 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:24:10 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T20:51:50.334 INFO:teuthology.orchestra.run.smithi070.stdout:Package ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T20:51:50.526 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T20:51:50.527 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2023-10-05T20:51:50.527 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T20:51:50.872 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install cephadm 2023-10-05T20:51:51.723 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:24:13 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T20:51:53.304 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T20:51:53.305 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:51:53.305 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-10-05T20:51:53.305 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:51:53.305 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T20:51:53.305 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 80 k 2023-10-05T20:51:53.305 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:51:53.305 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T20:51:53.305 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:51:53.305 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2023-10-05T20:51:53.305 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:51:53.306 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 80 k 2023-10-05T20:51:53.306 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 364 k 2023-10-05T20:51:53.306 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T20:51:53.467 INFO:teuthology.orchestra.run.smithi070.stdout:cephadm-17.2.6-1278.g487ffcb5.el8.noarch.rpm 500 kB/s | 80 kB 00:00 2023-10-05T20:51:53.468 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:51:53.468 INFO:teuthology.orchestra.run.smithi070.stdout:Total 496 kB/s | 80 kB 00:00 2023-10-05T20:51:53.468 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T20:51:53.475 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T20:51:53.475 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T20:51:53.477 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T20:51:53.480 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T20:51:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T20:51:53.965 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T20:51:54.098 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T20:51:54.322 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T20:51:54.948 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T20:51:54.948 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:51:54.948 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T20:51:54.948 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:51:54.948 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:51:54.948 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T20:51:55.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-immutable-object-cache 2023-10-05T20:51:56.027 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:24:18 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T20:51:57.604 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T20:51:57.604 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:51:57.605 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2023-10-05T20:51:57.605 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:51:57.605 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T20:51:57.605 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 162 k 2023-10-05T20:51:57.605 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:51:57.605 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T20:51:57.605 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:51:57.605 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2023-10-05T20:51:57.605 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:51:57.606 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 162 k 2023-10-05T20:51:57.606 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 490 k 2023-10-05T20:51:57.606 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T20:51:57.896 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-immutable-object-cache-17.2.6-1278.g487ffc 557 kB/s | 162 kB 00:00 2023-10-05T20:51:57.897 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:51:57.897 INFO:teuthology.orchestra.run.smithi070.stdout:Total 554 kB/s | 162 kB 00:00 2023-10-05T20:51:57.897 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T20:51:57.907 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T20:51:57.907 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T20:51:57.954 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T20:51:57.957 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T20:51:58.077 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T20:51:58.214 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5. 1/1 2023-10-05T20:51:58.620 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5. 1/1 2023-10-05T20:51:59.247 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5. 1/1 2023-10-05T20:51:59.247 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:51:59.248 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T20:51:59.248 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:51:59.248 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:51:59.248 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T20:51:59.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr 2023-10-05T20:52:00.301 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:24:22 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T20:52:01.697 INFO:teuthology.orchestra.run.smithi070.stdout:Package ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T20:52:01.884 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T20:52:01.885 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2023-10-05T20:52:01.886 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T20:52:02.045 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-dashboard 2023-10-05T20:52:02.899 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:24:24 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T20:52:04.478 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T20:52:04.479 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:52:04.479 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-10-05T20:52:04.479 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:52:04.479 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T20:52:04.479 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 1.7 M 2023-10-05T20:52:04.479 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2023-10-05T20:52:04.479 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 23 k 2023-10-05T20:52:04.479 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 14 k 2023-10-05T20:52:04.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-05T20:52:04.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-05T20:52:04.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-05T20:52:04.480 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:52:04.480 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T20:52:04.480 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:52:04.480 INFO:teuthology.orchestra.run.smithi070.stdout:Install 6 Packages 2023-10-05T20:52:04.480 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:52:04.480 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 2.0 M 2023-10-05T20:52:04.481 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 8.0 M 2023-10-05T20:52:04.481 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T20:52:04.824 INFO:teuthology.orchestra.run.smithi070.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1278.g487 166 kB/s | 23 kB 00:00 2023-10-05T20:52:04.883 INFO:teuthology.orchestra.run.smithi070.stdout:(2/6): python3-jwt-1.6.1-2.el8.noarch.rpm 731 kB/s | 43 kB 00:00 2023-10-05T20:52:04.933 INFO:teuthology.orchestra.run.smithi070.stdout:(3/6): ceph-prometheus-alerts-17.2.6-1278.g487f 58 kB/s | 14 kB 00:00 2023-10-05T20:52:05.300 INFO:teuthology.orchestra.run.smithi070.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 81 kB/s | 34 kB 00:00 2023-10-05T20:52:05.392 INFO:teuthology.orchestra.run.smithi070.stdout:(5/6): ceph-mgr-dashboard-17.2.6-1278.g487ffcb5 2.4 MB/s | 1.7 MB 00:00 2023-10-05T20:52:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 298 kB/s | 196 kB 00:00 2023-10-05T20:52:05.595 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:52:05.595 INFO:teuthology.orchestra.run.smithi070.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2023-10-05T20:52:05.606 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T20:52:05.616 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T20:52:05.616 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T20:52:05.695 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T20:52:05.698 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T20:52:05.968 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T20:52:06.160 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-10-05T20:52:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-10-05T20:52:06.548 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-10-05T20:52:06.668 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1278.g487ffcb5.el8.n 4/6 2023-10-05T20:52:06.958 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1278.g487ffcb5.el8. 5/6 2023-10-05T20:52:07.133 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarc 6/6 2023-10-05T20:52:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarc 6/6 2023-10-05T20:52:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1278.g487ffcb5.el8. 1/6 2023-10-05T20:52:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarc 2/6 2023-10-05T20:52:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1278.g487ffcb5.el8.n 3/6 2023-10-05T20:52:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-10-05T20:52:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-10-05T20:52:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-10-05T20:52:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:52:08.073 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T20:52:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:52:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:52:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:52:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-05T20:52:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-05T20:52:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-05T20:52:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:52:08.074 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T20:52:08.187 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-05T20:52:09.034 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:24:31 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T20:52:10.612 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T20:52:10.613 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================================ 2023-10-05T20:52:10.614 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-10-05T20:52:10.614 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================================ 2023-10-05T20:52:10.614 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T20:52:10.614 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 7.4 M 2023-10-05T20:52:10.614 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2023-10-05T20:52:10.614 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-10-05T20:52:10.614 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-10-05T20:52:10.614 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-05T20:52:10.614 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-05T20:52:10.614 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-05T20:52:10.614 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-05T20:52:10.615 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-05T20:52:10.615 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:52:10.615 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T20:52:10.615 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================================ 2023-10-05T20:52:10.615 INFO:teuthology.orchestra.run.smithi070.stdout:Install 8 Packages 2023-10-05T20:52:10.615 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:52:10.616 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 35 M 2023-10-05T20:52:10.616 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 210 M 2023-10-05T20:52:10.616 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T20:52:11.803 INFO:teuthology.orchestra.run.smithi070.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.6-127 9.6 MB/s | 7.4 MB 00:00 2023-10-05T20:57:11.495 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] openblas-0.3.15-4.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/openblas-0.3.15-4.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:57:11.495 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] openblas-threads-0.3.15-4.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/openblas-threads-0.3.15-4.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:57:12.628 INFO:teuthology.orchestra.run.smithi070.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 16 kB/s | 4.7 MB 05:01 2023-10-05T20:57:12.704 INFO:teuthology.orchestra.run.smithi070.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 16 kB/s | 4.9 MB 05:01 2023-10-05T20:57:12.746 INFO:teuthology.orchestra.run.smithi070.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2023-10-05T20:57:12.912 INFO:teuthology.orchestra.run.smithi070.stdout:(5/8): libgfortran-8.5.0-20.el8.x86_64.rpm 3.8 MB/s | 645 kB 00:00 2023-10-05T20:57:12.946 INFO:teuthology.orchestra.run.smithi070.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2023-10-05T20:57:13.421 INFO:teuthology.orchestra.run.smithi070.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 19 MB/s | 14 MB 00:00 2023-10-05T20:57:15.811 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-numpy-1.14.3-10.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-numpy-1.14.3-10.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:57:16.821 INFO:teuthology.orchestra.run.smithi070.stdout:(8/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 13 kB/s | 3.7 MB 05:05 2023-10-05T20:57:16.821 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:57:16.821 INFO:teuthology.orchestra.run.smithi070.stdout:Total 118 kB/s | 35 MB 05:06 2023-10-05T20:57:17.051 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T20:57:17.069 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T20:57:17.069 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T20:57:17.449 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T20:57:17.502 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T20:57:18.012 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T20:57:18.133 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-10-05T20:57:18.246 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-10-05T20:57:18.366 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-10-05T20:57:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-10-05T20:57:19.125 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-10-05T20:57:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-10-05T20:57:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-05T20:57:20.572 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-05T20:57:20.903 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-05T20:57:21.141 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-05T20:57:22.905 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-05T20:57:24.163 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-10-05T20:57:24.374 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb 8/8 2023-10-05T20:57:24.685 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb 8/8 2023-10-05T20:57:24.685 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb 1/8 2023-10-05T20:57:24.685 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-10-05T20:57:24.685 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-10-05T20:57:24.685 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-10-05T20:57:24.685 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-10-05T20:57:24.685 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-10-05T20:57:24.685 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-10-05T20:57:25.338 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-10-05T20:57:25.338 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:57:25.338 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T20:57:25.338 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:57:25.338 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-10-05T20:57:25.338 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-10-05T20:57:25.338 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-05T20:57:25.339 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-05T20:57:25.339 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-05T20:57:25.339 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-05T20:57:25.339 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-05T20:57:25.339 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:57:25.339 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T20:57:25.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-rook 2023-10-05T20:57:26.327 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:29:48 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T20:57:27.914 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T20:57:27.915 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:57:27.915 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-10-05T20:57:27.916 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:57:27.916 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T20:57:27.916 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 47 k 2023-10-05T20:57:27.916 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2023-10-05T20:57:27.916 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-05T20:57:27.916 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-05T20:57:27.916 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-05T20:57:27.916 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-05T20:57:27.916 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-05T20:57:27.916 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-05T20:57:27.916 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-05T20:57:27.917 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-05T20:57:27.917 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-05T20:57:27.917 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-05T20:57:27.917 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-05T20:57:27.917 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-05T20:57:27.917 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:57:27.917 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T20:57:27.917 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T20:57:27.917 INFO:teuthology.orchestra.run.smithi070.stdout:Install 13 Packages 2023-10-05T20:57:27.917 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T20:57:27.918 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 1.9 M 2023-10-05T20:57:27.918 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 25 M 2023-10-05T20:57:27.918 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T20:57:28.668 INFO:teuthology.orchestra.run.smithi070.stdout:(1/13): ceph-mgr-rook-17.2.6-1278.g487ffcb5.el8 138 kB/s | 47 kB 00:00 2023-10-05T21:01:35.198 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-jsonpointer-1.10-11.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-1.10-11.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T21:01:35.199 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-jsonpatch-1.21-2.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-2.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T21:01:35.409 INFO:teuthology.orchestra.run.smithi070.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 83 B/s | 20 kB 04:07 2023-10-05T21:01:35.435 INFO:teuthology.orchestra.run.smithi070.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 110 B/s | 27 kB 04:07 2023-10-05T21:01:35.501 INFO:teuthology.orchestra.run.smithi070.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.3 MB/s | 155 kB 00:00 2023-10-05T21:01:35.535 INFO:teuthology.orchestra.run.smithi070.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.2 MB/s | 43 kB 00:00 2023-10-05T21:01:35.786 INFO:teuthology.orchestra.run.smithi070.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 132 kB/s | 33 kB 00:00 2023-10-05T21:01:35.861 INFO:teuthology.orchestra.run.smithi070.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 218 kB/s | 16 kB 00:00 2023-10-05T21:01:35.986 INFO:teuthology.orchestra.run.smithi070.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 650 kB/s | 81 kB 00:00 2023-10-05T21:01:36.136 INFO:teuthology.orchestra.run.smithi070.stdout:(9/13): python3-kubernetes-11.0.0-6.el8.noarch. 7.8 MB/s | 1.2 MB 00:00 2023-10-05T21:01:36.220 INFO:teuthology.orchestra.run.smithi070.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 769 kB/s | 64 kB 00:00 2023-10-05T21:01:36.304 INFO:teuthology.orchestra.run.smithi070.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 734 kB/s | 61 kB 00:00 2023-10-05T21:01:51.654 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-pyasn1-0.3.7-6.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.3.7-6.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T21:01:51.951 INFO:teuthology.orchestra.run.smithi070.stdout:(12/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 491 B/s | 126 kB 04:23 2023-10-05T21:05:45.055 INFO:teuthology.orchestra.run.smithi070.stdout:[MIRROR] python3-pyasn1-modules-0.3.7-6.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pyasn1-modules-0.3.7-6.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T21:05:45.248 INFO:teuthology.orchestra.run.smithi070.stdout:(13/13): python3-pyasn1-modules-0.3.7-6.el8.noa 451 B/s | 110 kB 04:09 2023-10-05T21:05:45.249 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T21:05:45.249 INFO:teuthology.orchestra.run.smithi070.stdout:Total 4.0 kB/s | 1.9 MB 08:17 2023-10-05T21:05:45.302 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T21:05:45.318 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T21:05:45.318 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T21:05:45.404 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T21:05:45.407 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T21:05:45.886 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T21:05:46.154 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-05T21:05:46.344 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-10-05T21:05:46.562 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-10-05T21:05:46.775 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-10-05T21:05:46.974 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-10-05T21:05:47.198 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-10-05T21:05:47.465 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-10-05T21:05:47.698 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-05T21:05:48.736 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-05T21:05:49.008 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-10-05T21:05:49.192 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-05T21:05:49.385 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-10-05T21:05:49.543 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 13/13 2023-10-05T21:05:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 13/13 2023-10-05T21:05:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 1/13 2023-10-05T21:05:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-10-05T21:05:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-10-05T21:05:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-10-05T21:05:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-10-05T21:05:49.823 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-05T21:05:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-05T21:05:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-10-05T21:05:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-05T21:05:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-10-05T21:05:49.824 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-10-05T21:05:49.825 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-10-05T21:05:50.513 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-10-05T21:05:50.513 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:05:50.513 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T21:05:50.514 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T21:05:50.514 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-05T21:05:50.514 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-05T21:05:50.514 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-05T21:05:50.514 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-05T21:05:50.514 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-05T21:05:50.514 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-05T21:05:50.514 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-05T21:05:50.514 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-05T21:05:50.514 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-05T21:05:50.515 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-05T21:05:50.515 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-05T21:05:50.515 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-05T21:05:50.515 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:05:50.515 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:05:50.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-cephadm 2023-10-05T21:05:51.481 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:13 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:05:53.063 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:05:53.064 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:05:53.064 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-10-05T21:05:53.064 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:05:53.064 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T21:05:53.064 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 121 k 2023-10-05T21:05:53.064 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2023-10-05T21:05:53.064 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-05T21:05:53.064 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-05T21:05:53.064 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-05T21:05:53.064 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-05T21:05:53.064 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:05:53.064 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T21:05:53.065 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:05:53.065 INFO:teuthology.orchestra.run.smithi070.stdout:Install 5 Packages 2023-10-05T21:05:53.065 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:05:53.065 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 6.0 M 2023-10-05T21:05:53.065 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 26 M 2023-10-05T21:05:53.065 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T21:05:53.614 INFO:teuthology.orchestra.run.smithi070.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1278.g487ffcb5.e 408 kB/s | 121 kB 00:00 2023-10-05T21:05:53.822 INFO:teuthology.orchestra.run.smithi070.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.2 MB/s | 476 kB 00:00 2023-10-05T21:05:53.873 INFO:teuthology.orchestra.run.smithi070.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 1.2 MB/s | 60 kB 00:00 2023-10-05T21:05:53.906 INFO:teuthology.orchestra.run.smithi070.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 913 kB/s | 538 kB 00:00 2023-10-05T21:05:55.182 INFO:teuthology.orchestra.run.smithi070.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 2.6 MB/s | 4.8 MB 00:01 2023-10-05T21:05:55.184 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T21:05:55.184 INFO:teuthology.orchestra.run.smithi070.stdout:Total 2.8 MB/s | 6.0 MB 00:02 2023-10-05T21:05:55.238 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T21:05:55.250 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T21:05:55.250 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T21:05:55.352 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T21:05:55.355 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T21:05:55.653 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T21:05:55.947 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-10-05T21:05:56.905 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-05T21:05:57.263 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-10-05T21:05:57.518 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-05T21:05:57.683 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 5/5 2023-10-05T21:05:57.966 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 5/5 2023-10-05T21:05:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/5 2023-10-05T21:05:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-10-05T21:05:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-10-05T21:05:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-05T21:05:58.589 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-10-05T21:05:58.590 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:05:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T21:05:58.590 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T21:05:58.590 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-05T21:05:58.590 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-05T21:05:58.590 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-05T21:05:58.590 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-05T21:05:58.590 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:05:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:05:58.695 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-fuse 2023-10-05T21:05:59.546 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:21 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:01.133 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:01.133 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:01.133 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-10-05T21:06:01.134 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:01.134 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T21:06:01.134 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 876 k 2023-10-05T21:06:01.134 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2023-10-05T21:06:01.134 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-05T21:06:01.134 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:01.134 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T21:06:01.134 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:01.134 INFO:teuthology.orchestra.run.smithi070.stdout:Install 2 Packages 2023-10-05T21:06:01.134 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:01.135 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 959 k 2023-10-05T21:06:01.135 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 3.0 M 2023-10-05T21:06:01.135 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T21:06:01.255 INFO:teuthology.orchestra.run.smithi070.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 1.0 MB/s | 83 kB 00:00 2023-10-05T21:06:02.131 INFO:teuthology.orchestra.run.smithi070.stdout:(2/2): ceph-fuse-17.2.6-1278.g487ffcb5.el8.x86_ 917 kB/s | 876 kB 00:00 2023-10-05T21:06:02.131 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T21:06:02.131 INFO:teuthology.orchestra.run.smithi070.stdout:Total 962 kB/s | 959 kB 00:00 2023-10-05T21:06:02.142 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T21:06:02.164 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T21:06:02.164 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T21:06:02.249 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T21:06:02.252 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T21:06:02.417 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T21:06:02.657 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-10-05T21:06:02.840 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/2 2023-10-05T21:06:03.224 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/2 2023-10-05T21:06:03.224 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/2 2023-10-05T21:06:03.848 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-10-05T21:06:03.849 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:03.849 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T21:06:03.849 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-10-05T21:06:03.849 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:03.849 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:04.062 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-volume 2023-10-05T21:06:04.920 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:26 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:06.505 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:06.506 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:06.506 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-10-05T21:06:06.506 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:06.506 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T21:06:06.507 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 267 k 2023-10-05T21:06:06.507 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:06.507 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T21:06:06.507 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:06.507 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2023-10-05T21:06:06.507 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:06.507 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 267 k 2023-10-05T21:06:06.508 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 1.3 M 2023-10-05T21:06:06.508 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T21:06:06.719 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-volume-17.2.6-1278.g487ffcb5.el8.noarch.rp 1.2 MB/s | 267 kB 00:00 2023-10-05T21:06:06.719 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T21:06:06.720 INFO:teuthology.orchestra.run.smithi070.stdout:Total 1.2 MB/s | 267 kB 00:00 2023-10-05T21:06:06.720 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T21:06:06.730 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T21:06:06.730 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T21:06:06.765 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T21:06:06.768 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T21:06:06.943 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T21:06:07.164 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T21:06:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T21:06:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T21:06:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:08.200 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T21:06:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T21:06:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:08.200 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:08.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librados-devel 2023-10-05T21:06:09.266 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:31 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:10.854 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:10.854 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2023-10-05T21:06:10.855 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:10.855 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T21:06:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 130 k 2023-10-05T21:06:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:10.855 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T21:06:10.855 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:10.855 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2023-10-05T21:06:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:10.856 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 130 k 2023-10-05T21:06:10.856 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 464 k 2023-10-05T21:06:10.856 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T21:06:11.164 INFO:teuthology.orchestra.run.smithi070.stdout:librados-devel-17.2.6-1278.g487ffcb5.el8.x86_64 421 kB/s | 130 kB 00:00 2023-10-05T21:06:11.165 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T21:06:11.165 INFO:teuthology.orchestra.run.smithi070.stdout:Total 419 kB/s | 130 kB 00:00 2023-10-05T21:06:11.165 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T21:06:11.170 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T21:06:11.171 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T21:06:11.219 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T21:06:11.222 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T21:06:11.330 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T21:06:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:11.850 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:12.485 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:12.485 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:12.485 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T21:06:12.485 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T21:06:12.485 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:12.485 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:12.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install libcephfs2 2023-10-05T21:06:13.550 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:35 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:14.926 INFO:teuthology.orchestra.run.smithi070.stdout:Package libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T21:06:15.116 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:15.118 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2023-10-05T21:06:15.118 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:15.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install libcephfs-devel 2023-10-05T21:06:16.079 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:38 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:17.654 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:17.655 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:17.655 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2023-10-05T21:06:17.655 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:17.655 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T21:06:17.655 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 25 k 2023-10-05T21:06:17.655 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:17.655 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T21:06:17.655 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:17.655 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2023-10-05T21:06:17.655 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:17.656 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 25 k 2023-10-05T21:06:17.656 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 109 k 2023-10-05T21:06:17.656 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T21:06:17.809 INFO:teuthology.orchestra.run.smithi070.stdout:libcephfs-devel-17.2.6-1278.g487ffcb5.el8.x86_6 162 kB/s | 25 kB 00:00 2023-10-05T21:06:17.810 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T21:06:17.810 INFO:teuthology.orchestra.run.smithi070.stdout:Total 161 kB/s | 25 kB 00:00 2023-10-05T21:06:17.810 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T21:06:17.813 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T21:06:17.813 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T21:06:17.815 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T21:06:17.818 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T21:06:17.871 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T21:06:18.092 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:18.345 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:18.933 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T21:06:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T21:06:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:18.934 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:19.050 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librados2 2023-10-05T21:06:19.906 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:41 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:21.291 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T21:06:21.480 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:21.481 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2023-10-05T21:06:21.481 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:21.583 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librbd1 2023-10-05T21:06:22.436 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:44 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:23.827 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T21:06:24.019 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:24.020 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2023-10-05T21:06:24.020 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:24.125 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rados 2023-10-05T21:06:24.978 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:46 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:26.376 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T21:06:26.566 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:26.567 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2023-10-05T21:06:26.568 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:26.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rgw 2023-10-05T21:06:27.577 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:49 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:28.957 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T21:06:29.151 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:29.152 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2023-10-05T21:06:29.153 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:29.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-cephfs 2023-10-05T21:06:30.129 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:52 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:31.524 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T21:06:31.711 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:31.712 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2023-10-05T21:06:31.713 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:31.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rbd 2023-10-05T21:06:32.792 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:54 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:34.191 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T21:06:34.380 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:34.382 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2023-10-05T21:06:34.382 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:34.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-fuse 2023-10-05T21:06:35.401 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:38:57 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:36.997 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:36.997 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:36.998 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2023-10-05T21:06:36.998 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:36.998 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T21:06:36.998 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 90 k 2023-10-05T21:06:36.998 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:36.998 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T21:06:36.998 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:36.998 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2023-10-05T21:06:36.998 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:36.999 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 90 k 2023-10-05T21:06:36.999 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 264 k 2023-10-05T21:06:36.999 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T21:06:37.357 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-fuse-17.2.6-1278.g487ffcb5.el8.x86_64.rpm 252 kB/s | 90 kB 00:00 2023-10-05T21:06:37.358 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T21:06:37.358 INFO:teuthology.orchestra.run.smithi070.stdout:Total 251 kB/s | 90 kB 00:00 2023-10-05T21:06:37.358 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T21:06:37.368 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T21:06:37.368 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T21:06:37.417 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T21:06:37.420 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T21:06:37.527 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T21:06:37.698 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:37.970 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:38.577 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:38.577 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:38.577 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T21:06:38.578 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T21:06:38.578 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:38.578 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:38.785 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-mirror 2023-10-05T21:06:39.638 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:39:01 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:41.228 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:41.229 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2023-10-05T21:06:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T21:06:41.229 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.5 M 2023-10-05T21:06:41.229 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T21:06:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2023-10-05T21:06:41.229 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 3.5 M 2023-10-05T21:06:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 16 M 2023-10-05T21:06:41.230 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T21:06:41.676 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-mirror-17.2.6-1278.g487ffcb5.el8.x86_64.rpm 7.7 MB/s | 3.5 MB 00:00 2023-10-05T21:06:41.677 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T21:06:41.677 INFO:teuthology.orchestra.run.smithi070.stdout:Total 7.7 MB/s | 3.5 MB 00:00 2023-10-05T21:06:41.677 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T21:06:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T21:06:41.689 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T21:06:41.752 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T21:06:41.755 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T21:06:42.201 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T21:06:42.415 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:42.823 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:43.440 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:43.440 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:43.440 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T21:06:43.440 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T21:06:43.440 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:43.440 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:43.642 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-nbd 2023-10-05T21:06:44.504 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:39:06 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:06:46.086 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:06:46.086 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:46.086 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2023-10-05T21:06:46.086 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:46.087 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T21:06:46.087 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 173 k 2023-10-05T21:06:46.087 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:46.087 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T21:06:46.087 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:06:46.087 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2023-10-05T21:06:46.087 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:46.088 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 173 k 2023-10-05T21:06:46.088 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 552 k 2023-10-05T21:06:46.088 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T21:06:46.455 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-nbd-17.2.6-1278.g487ffcb5.el8.x86_64.rpm 471 kB/s | 173 kB 00:00 2023-10-05T21:06:46.456 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T21:06:46.456 INFO:teuthology.orchestra.run.smithi070.stdout:Total 469 kB/s | 173 kB 00:00 2023-10-05T21:06:46.456 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T21:06:46.466 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T21:06:46.466 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T21:06:46.516 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T21:06:46.519 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T21:06:46.643 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T21:06:46.795 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:47.088 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:47.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T21:06:47.731 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:47.731 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T21:06:47.731 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T21:06:47.731 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:47.731 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:06:47.924 DEBUG:teuthology.parallel:result is None 2023-10-05T21:06:47.925 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T21:06:48.147 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-05T21:06:48.169 INFO:teuthology.orchestra.run.smithi070.stdout:17.2.6-1278.g487ffcb5.el8 2023-10-05T21:06:48.169 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1278.g487ffcb5.el8 2023-10-05T21:06:48.170 INFO:teuthology.task.install:The correct ceph version 17.2.6-1278.g487ffcb5 is installed. 2023-10-05T21:06:48.170 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-05T21:06:48.170 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:48.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-05T21:06:48.238 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-05T21:06:48.238 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:48.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2023-10-05T21:06:48.304 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-05T21:06:48.371 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-05T21:06:48.372 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:48.372 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-05T21:06:48.438 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-05T21:06:48.505 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-05T21:06:48.506 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:48.506 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2023-10-05T21:06:48.572 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-05T21:06:48.640 INFO:teuthology.run_tasks:Running task ceph... 2023-10-05T21:06:48.702 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-05T21:06:48.702 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /var/log/ceph 2023-10-05T21:06:48.729 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-05T21:06:48.730 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-05T21:06:48.797 INFO:tasks.ceph:Creating extra log directories... 2023-10-05T21:06:48.797 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-05T21:06:48.872 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-05T21:06:48.872 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt sse s3 backend': 'vault', 'rgw crypt sse s3 vault auth': 'token', 'rgw crypt sse s3 vault prefix': '/v1/transit/', 'rgw crypt sse s3 vault secret engine': 'transit', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-05T21:06:48.872 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411774', 'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7411774', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt sse s3 backend': 'vault', 'rgw crypt sse s3 vault auth': 'token', 'rgw crypt sse s3 vault prefix': '/v1/transit/', 'rgw crypt sse s3 vault secret engine': 'transit', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}}, 'rgw': {'client.0': {'use-vault-role': 'client.0'}}, 's3tests': {'with-sse-s3': True}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 880, 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'targets': {'smithi070.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8z/vefdvZUw41n3xNDmfIeVLkjdixVzTHNkrvMPaVIX3CjFH+aUP2pxblFIoZ876J04OLGO6n7SqTluvifork='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'vault': {'client.0': {'engine': 'transit', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/transit/keys/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1'}, {'path': 'my-key-2'}]}}}, {'rgw': {'client.0': None}}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-05_14:30:58', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-05T21:06:48.873 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-05T21:06:48.923 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/run/ceph 2023-10-05T21:06:48.990 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:48.990 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2023-10-05T21:06:49.048 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-05T21:06:49.048 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2023-10-05T21:06:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-05T21:06:49.106 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T21:06:49.106 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 6h/6d Inode: 392192 Links: 1 2023-10-05T21:06:49.106 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T21:06:49.106 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T21:06:49.106 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2023-10-05 21:06:42.570896031 +0000 2023-10-05T21:06:49.106 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2023-10-05 20:12:37.058334295 +0000 2023-10-05T21:06:49.106 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2023-10-05 20:12:37.058334295 +0000 2023-10-05T21:06:49.107 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2023-10-05T21:06:49.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-05T21:06:49.171 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2023-10-05T21:06:49.171 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2023-10-05T21:06:49.172 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000110572 s, 4.6 MB/s 2023-10-05T21:06:49.172 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-05T21:06:49.232 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2023-10-05T21:06:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-05T21:06:49.290 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T21:06:49.290 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 6h/6d Inode: 398421 Links: 1 2023-10-05T21:06:49.290 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T21:06:49.290 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T21:06:49.290 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2023-10-05 21:06:42.570896031 +0000 2023-10-05T21:06:49.291 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2023-10-05 20:12:37.561319878 +0000 2023-10-05T21:06:49.291 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2023-10-05 20:12:37.561319878 +0000 2023-10-05T21:06:49.291 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2023-10-05T21:06:49.291 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-05T21:06:49.356 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2023-10-05T21:06:49.356 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2023-10-05T21:06:49.356 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 8.1753e-05 s, 6.3 MB/s 2023-10-05T21:06:49.357 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-05T21:06:49.415 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2023-10-05T21:06:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-05T21:06:49.474 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T21:06:49.474 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 6h/6d Inode: 399496 Links: 1 2023-10-05T21:06:49.474 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T21:06:49.474 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T21:06:49.474 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2023-10-05 21:06:42.570896031 +0000 2023-10-05T21:06:49.475 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2023-10-05 20:12:38.067305375 +0000 2023-10-05T21:06:49.475 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2023-10-05 20:12:38.067305375 +0000 2023-10-05T21:06:49.475 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2023-10-05T21:06:49.475 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-05T21:06:49.539 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2023-10-05T21:06:49.539 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2023-10-05T21:06:49.539 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000101204 s, 5.1 MB/s 2023-10-05T21:06:49.540 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-05T21:06:49.599 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2023-10-05T21:06:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-05T21:06:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T21:06:49.658 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 6h/6d Inode: 397086 Links: 1 2023-10-05T21:06:49.658 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T21:06:49.658 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T21:06:49.658 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2023-10-05 21:06:42.570896031 +0000 2023-10-05T21:06:49.658 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2023-10-05 20:12:38.564291130 +0000 2023-10-05T21:06:49.658 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2023-10-05 20:12:38.564291130 +0000 2023-10-05T21:06:49.658 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2023-10-05T21:06:49.659 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-05T21:06:49.723 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2023-10-05T21:06:49.723 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2023-10-05T21:06:49.723 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000105319 s, 4.9 MB/s 2023-10-05T21:06:49.724 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-05T21:06:49.784 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T21:06:49.784 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-05T21:06:49.784 INFO:tasks.ceph:Generating config... 2023-10-05T21:06:49.785 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-05T21:06:49.785 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-05T21:06:49.785 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = vault 2023-10-05T21:06:49.785 INFO:tasks.ceph:[client] rgw crypt sse s3 backend = vault 2023-10-05T21:06:49.785 INFO:tasks.ceph:[client] rgw crypt sse s3 vault auth = token 2023-10-05T21:06:49.785 INFO:tasks.ceph:[client] rgw crypt sse s3 vault prefix = /v1/transit/ 2023-10-05T21:06:49.786 INFO:tasks.ceph:[client] rgw crypt sse s3 vault secret engine = transit 2023-10-05T21:06:49.786 INFO:tasks.ceph:[client] rgw crypt vault auth = token 2023-10-05T21:06:49.786 INFO:tasks.ceph:[client] rgw crypt vault prefix = /v1/transit/ 2023-10-05T21:06:49.786 INFO:tasks.ceph:[client] rgw crypt vault secret engine = transit 2023-10-05T21:06:49.786 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-05T21:06:49.786 INFO:tasks.ceph:[client] setuser = ceph 2023-10-05T21:06:49.786 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-05T21:06:49.786 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-05T21:06:49.786 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-05T21:06:49.786 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-05T21:06:49.786 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-05T21:06:49.786 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-05T21:06:49.787 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-05T21:06:49.787 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-05T21:06:49.787 INFO:tasks.ceph:Setting up mon.a... 2023-10-05T21:06:49.787 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-05T21:06:49.882 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.keyring 2023-10-05T21:06:49.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-05T21:06:49.925 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T21:06:49.992 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.70')] 2023-10-05T21:06:49.992 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.70'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt sse s3 backend': 'vault', 'rgw crypt sse s3 vault auth': 'token', 'rgw crypt sse s3 vault prefix': '/v1/transit/', 'rgw crypt sse s3 vault secret engine': 'transit', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2023-10-05T21:06:49.993 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:49.993 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-05T21:06:50.050 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.70 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-05T21:06:50.130 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setuser ceph since I am not root 2023-10-05T21:06:50.130 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T21:06:50.131 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-05T21:06:50.131 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: generated fsid 18ea46c7-744d-428a-a77f-8692a242600e 2023-10-05T21:06:50.131 INFO:teuthology.orchestra.run.smithi070.stdout:setting min_mon_release = octopus 2023-10-05T21:06:50.131 INFO:teuthology.orchestra.run.smithi070.stdout:epoch 0 2023-10-05T21:06:50.131 INFO:teuthology.orchestra.run.smithi070.stdout:fsid 18ea46c7-744d-428a-a77f-8692a242600e 2023-10-05T21:06:50.132 INFO:teuthology.orchestra.run.smithi070.stdout:last_changed 2023-10-05T21:06:50.130762+0000 2023-10-05T21:06:50.132 INFO:teuthology.orchestra.run.smithi070.stdout:created 2023-10-05T21:06:50.130762+0000 2023-10-05T21:06:50.132 INFO:teuthology.orchestra.run.smithi070.stdout:min_mon_release 15 (octopus) 2023-10-05T21:06:50.132 INFO:teuthology.orchestra.run.smithi070.stdout:election_strategy: 1 2023-10-05T21:06:50.132 INFO:teuthology.orchestra.run.smithi070.stdout:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2023-10-05T21:06:50.132 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-05T21:06:50.133 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-05T21:06:50.191 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 18ea46c7-744d-428a-a77f-8692a242600e... 2023-10-05T21:06:50.192 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-05T21:06:50.275 INFO:teuthology.orchestra.run.smithi070.stdout:[global] 2023-10-05T21:06:50.275 INFO:teuthology.orchestra.run.smithi070.stdout: chdir = "" 2023-10-05T21:06:50.276 INFO:teuthology.orchestra.run.smithi070.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-05T21:06:50.276 INFO:teuthology.orchestra.run.smithi070.stdout: auth supported = cephx 2023-10-05T21:06:50.276 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.276 INFO:teuthology.orchestra.run.smithi070.stdout: filestore xattr use omap = true 2023-10-05T21:06:50.276 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.276 INFO:teuthology.orchestra.run.smithi070.stdout: mon clock drift allowed = 1.000 2023-10-05T21:06:50.276 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.276 INFO:teuthology.orchestra.run.smithi070.stdout: osd crush chooseleaf type = 0 2023-10-05T21:06:50.276 INFO:teuthology.orchestra.run.smithi070.stdout: auth debug = true 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on old message = true 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on bug = true 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: mon pg warn max object skew = 0 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: # disable pg_autoscaler by default for new pools 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default size = 2 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.277 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow primary affinity = true 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow pg remap = true 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on legacy crush tunables = false 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on crush straw calc version zero = false 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on no sortbitwise = false 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on osd down out interval zero = false 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on too few osds = false 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: mon_allow_pool_size_one = true 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.278 INFO:teuthology.orchestra.run.smithi070.stdout: osd default data pool replay window = 5 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout: mon allow pool delete = true 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout: mon cluster log file level = debug 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout: debug asserts on shutdown = true 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout: mon health detail to clog = false 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout: mon host = 172.21.15.70 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout: fsid = 18ea46c7-744d-428a-a77f-8692a242600e 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout:[osd] 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout: osd journal size = 100 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.279 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub load threshold = 5.0 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub max interval = 600 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: osd mclock profile = high_recovery_ops 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: osd recover clone overlap = true 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: osd recovery max chunk = 1048576 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug shutdown = true 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug op order = true 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify stray on activate = true 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: osd open classes on start = true 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug pg log writeout = true 2023-10-05T21:06:50.280 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: osd deep scrub update digest min age = 30 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: osd map max advance = 10 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: journal zero on create = true 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: filestore ondisk finisher threads = 3 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: filestore apply finisher threads = 3 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: bdev debug aio = true 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug misdirected ops = true 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2023-10-05T21:06:50.281 INFO:teuthology.orchestra.run.smithi070.stdout: debug osd = 20 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout: osd shutdown pgref assert = True 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr] 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout: debug mgr = 20 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout: mgr/telemetry/nag = false 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout:[mon] 2023-10-05T21:06:50.282 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: debug paxos = 20 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: mon data avail warn = 5 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: mon mgr mkfs grace = 240 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd reporter subtree level = osd 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd prime pg temp = true 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: auth mon ticket ttl = 660 # 11m 2023-10-05T21:06:50.283 INFO:teuthology.orchestra.run.smithi070.stdout: auth service ticket ttl = 240 # 4m 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout: # don't complain about insecure global_id in the test suite 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout: # 1m isn't quite enough 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout: mon_down_mkfs_grace = 2m 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_filestore_osds = false 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout:[client] 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout: rgw cache enabled = true 2023-10-05T21:06:50.284 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable ops log = true 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable usage log = true 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: debug rgw = 20 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: rgw crypt require ssl = False 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: rgw crypt s3 kms backend = vault 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: rgw crypt sse s3 backend = vault 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: rgw crypt sse s3 vault auth = token 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: rgw crypt sse s3 vault prefix = /v1/transit/ 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: rgw crypt sse s3 vault secret engine = transit 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: rgw crypt vault auth = token 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: rgw crypt vault prefix = /v1/transit/ 2023-10-05T21:06:50.285 INFO:teuthology.orchestra.run.smithi070.stdout: rgw crypt vault secret engine = transit 2023-10-05T21:06:50.286 INFO:teuthology.orchestra.run.smithi070.stdout: setgroup = ceph 2023-10-05T21:06:50.286 INFO:teuthology.orchestra.run.smithi070.stdout: setuser = ceph 2023-10-05T21:06:50.286 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.a] 2023-10-05T21:06:50.287 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-05T21:06:50.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-05T21:06:50.380 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-05T21:06:50.380 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:50.380 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-05T21:06:50.397 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:50.398 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-05T21:06:50.455 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi070.front.sepia.ceph.com 2023-10-05T21:06:50.455 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:50.456 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-05T21:06:50.456 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T21:06:50.532 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:50.532 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-05T21:06:50.590 INFO:tasks.ceph:Setting up mon nodes... 2023-10-05T21:06:50.591 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-05T21:06:50.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-05T21:06:50.681 INFO:teuthology.orchestra.run.smithi070.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-05T21:06:50.683 INFO:tasks.ceph:Setting up mds nodes... 2023-10-05T21:06:50.683 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-05T21:06:50.683 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-05T21:06:50.723 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-05T21:06:50.735 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-05T21:06:50.735 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-05T21:06:50.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-05T21:06:50.803 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T21:06:50.803 INFO:tasks.ceph:role: osd.0 2023-10-05T21:06:50.803 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com 2023-10-05T21:06:50.803 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-05T21:06:50.869 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T21:06:50.869 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T21:06:50.869 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T21:06:50.869 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-05T21:06:50.869 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T21:06:50.870 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2023-10-05T21:06:50.870 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T21:06:50.870 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T21:06:50.870 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T21:06:50.870 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T21:06:50.895 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2023-10-05T21:06:50.896 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2023-10-05T21:06:50.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-05T21:06:50.977 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-05T21:06:51.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-05T21:06:51.115 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T21:06:51.116 INFO:tasks.ceph:role: osd.1 2023-10-05T21:06:51.116 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com 2023-10-05T21:06:51.116 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-05T21:06:51.182 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T21:06:51.183 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T21:06:51.183 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T21:06:51.183 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-05T21:06:51.183 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T21:06:51.183 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2023-10-05T21:06:51.183 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T21:06:51.183 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T21:06:51.184 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T21:06:51.184 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T21:06:51.205 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2023-10-05T21:06:51.207 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2023-10-05T21:06:51.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-05T21:06:51.286 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-05T21:06:51.357 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-05T21:06:51.424 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T21:06:51.424 INFO:tasks.ceph:role: osd.2 2023-10-05T21:06:51.424 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com 2023-10-05T21:06:51.424 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-05T21:06:51.490 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T21:06:51.490 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T21:06:51.490 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T21:06:51.490 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-05T21:06:51.490 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T21:06:51.491 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2023-10-05T21:06:51.491 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T21:06:51.491 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T21:06:51.491 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T21:06:51.491 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T21:06:51.513 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2023-10-05T21:06:51.514 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2023-10-05T21:06:51.515 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-05T21:06:51.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-05T21:06:51.665 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T21:06:51.763 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:51.761+0000 7f61ecb77540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-05T21:06:51.763 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:51.761+0000 7f61ecb77540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-05T21:06:51.763 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:51.761+0000 7f61ecb77540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T21:06:51.764 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:51.761+0000 7f61ecb77540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T21:06:51.764 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:51.761+0000 7f61ecb77540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T21:06:51.764 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:51.761+0000 7f61ecb77540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-05T21:06:52.285 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:52.283+0000 7f61ecb77540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T21:06:53.046 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:53.044+0000 7f61ecb77540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T21:06:53.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-05T21:06:53.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T21:06:53.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:53.946+0000 7fa2ec42b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-05T21:06:53.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:53.947+0000 7fa2ec42b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-05T21:06:53.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:53.947+0000 7fa2ec42b540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T21:06:53.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:53.947+0000 7fa2ec42b540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T21:06:53.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:53.947+0000 7fa2ec42b540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T21:06:53.950 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:53.947+0000 7fa2ec42b540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-05T21:06:54.470 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:54.468+0000 7fa2ec42b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T21:06:55.232 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:55.230+0000 7fa2ec42b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T21:06:56.019 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-05T21:06:56.051 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T21:06:56.143 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:56.140+0000 7f2cd6827540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-05T21:06:56.143 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:56.140+0000 7f2cd6827540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-05T21:06:56.143 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:56.141+0000 7f2cd6827540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T21:06:56.143 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:56.141+0000 7f2cd6827540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T21:06:56.143 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:56.141+0000 7f2cd6827540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T21:06:56.143 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:56.141+0000 7f2cd6827540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-05T21:06:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:56.664+0000 7f2cd6827540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T21:06:57.428 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:06:57.426+0000 7f2cd6827540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T21:06:58.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-05T21:06:58.250 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-05T21:06:58.250 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:58.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-05T21:06:58.322 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:58.322 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-05T21:06:58.393 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:58.393 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-05T21:06:58.463 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:58.463 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-05T21:06:58.532 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:58.533 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-05T21:06:58.593 INFO:tasks.ceph:Adding keys to all mons... 2023-10-05T21:06:58.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-05T21:06:58.659 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.x] 2023-10-05T21:06:58.659 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBqJR9lWhGTKBAA3LShfBzI5+52b3wn7S37RQ== 2023-10-05T21:06:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.0] 2023-10-05T21:06:58.660 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBrJR9lS192LRAABqGdWB+3XXNxfA3gChWHeg== 2023-10-05T21:06:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.1] 2023-10-05T21:06:58.660 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBtJR9lK+6HOBAAWDFQh4OEOHpsPUrd+Poqiw== 2023-10-05T21:06:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.2] 2023-10-05T21:06:58.660 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBwJR9lz317CBAAdYZX22jVgCODOdnWmhNCew== 2023-10-05T21:06:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2023-10-05T21:06:58.660 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBqJR9lxCgfKxAAIdUIIiJ0v7MCYYSkDL84Lg== 2023-10-05T21:06:58.661 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-05T21:06:58.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T21:06:58.802 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T21:06:58.851 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T21:06:58.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-05T21:06:58.953 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-05T21:06:58.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-05T21:06:58.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-05T21:06:59.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-05T21:06:59.101 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-05T21:06:59.159 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-05T21:06:59.160 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-05T21:06:59.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-05T21:06:59.202 INFO:tasks.ceph.mon.a:Started 2023-10-05T21:06:59.202 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-05T21:06:59.202 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-05T21:06:59.202 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-05T21:06:59.204 INFO:tasks.ceph.mgr.x:Started 2023-10-05T21:06:59.204 INFO:tasks.ceph:Setting crush tunables to default 2023-10-05T21:06:59.204 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd crush tunables default 2023-10-05T21:06:59.404 INFO:teuthology.orchestra.run.smithi070.stderr:adjusted tunables profile to default 2023-10-05T21:06:59.415 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-05T21:06:59.415 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:59.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-05T21:06:59.441 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:59.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-05T21:06:59.508 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:06:59.508 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-05T21:06:59.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new 7481092a-4057-4980-8713-86e63cc546cb 0 2023-10-05T21:06:59.607 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:06:59.606+0000 7f8eed5821c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-05T21:06:59.778 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:06:59.777+0000 7f8eed5821c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-05T21:06:59.842 INFO:teuthology.orchestra.run.smithi070.stdout:0 2023-10-05T21:06:59.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new cc290433-0501-4fe0-85c0-a82ae80e3cea 1 2023-10-05T21:07:00.061 INFO:teuthology.orchestra.run.smithi070.stdout:1 2023-10-05T21:07:00.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new cf0ef326-b20b-48ad-82c7-6ff93ca6770c 2 2023-10-05T21:07:00.130 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:00.129+0000 7f8eed5821c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-05T21:07:00.280 INFO:teuthology.orchestra.run.smithi070.stdout:2 2023-10-05T21:07:00.292 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-05T21:07:00.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-05T21:07:00.295 INFO:tasks.ceph.osd.0:Started 2023-10-05T21:07:00.295 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-05T21:07:00.295 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-05T21:07:00.297 INFO:tasks.ceph.osd.1:Started 2023-10-05T21:07:00.297 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-05T21:07:00.297 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-05T21:07:00.299 INFO:tasks.ceph.osd.2:Started 2023-10-05T21:07:00.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T21:07:00.508 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:07:00.508 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":5,"fsid":"18ea46c7-744d-428a-a77f-8692a242600e","created":"2023-10-05T21:06:59.300586+0000","modified":"2023-10-05T21:07:00.278515+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T21:07:00.278515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"7481092a-4057-4980-8713-86e63cc546cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cc290433-0501-4fe0-85c0-a82ae80e3cea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cf0ef326-b20b-48ad-82c7-6ff93ca6770c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T21:07:00.520 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-05T21:07:00.521 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-05T21:07:00.675 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:00.674+0000 7f8eed5821c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-05T21:07:00.811 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:00.810+0000 7f8eed5821c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-05T21:07:00.822 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T21:07:00.895 INFO:tasks.ceph.osd.0.smithi070.stderr:2023-10-05T21:07:00.894+0000 7fdb43dd3540 -1 Falling back to public interface 2023-10-05T21:07:00.896 INFO:tasks.ceph.osd.1.smithi070.stderr:2023-10-05T21:07:00.895+0000 7ffae20d5540 -1 Falling back to public interface 2023-10-05T21:07:00.924 INFO:tasks.ceph.osd.2.smithi070.stderr:2023-10-05T21:07:00.923+0000 7f6a0ee5f540 -1 Falling back to public interface 2023-10-05T21:07:01.036 INFO:teuthology.misc.health.smithi070.stdout: 2023-10-05T21:07:01.036 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":5,"fsid":"18ea46c7-744d-428a-a77f-8692a242600e","created":"2023-10-05T21:06:59.300586+0000","modified":"2023-10-05T21:07:00.278515+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T21:07:00.278515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"7481092a-4057-4980-8713-86e63cc546cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cc290433-0501-4fe0-85c0-a82ae80e3cea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cf0ef326-b20b-48ad-82c7-6ff93ca6770c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T21:07:01.048 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-05T21:07:01.198 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:01.197+0000 7f8eed5821c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-05T21:07:01.911 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:01.910+0000 7f8eed5821c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-05T21:07:01.980 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:01.979+0000 7f8eed5821c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-05T21:07:02.113 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:02.112+0000 7f8eed5821c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-05T21:07:02.179 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:02.178+0000 7f8eed5821c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-05T21:07:02.357 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:02.356+0000 7f8eed5821c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-05T21:07:02.433 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:02.432+0000 7f8eed5821c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-05T21:07:02.507 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:02.506+0000 7f8eed5821c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-05T21:07:02.608 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:02.607+0000 7f8eed5821c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-05T21:07:02.782 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:02.781+0000 7f8eed5821c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-05T21:07:02.862 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:02.861+0000 7f8eed5821c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-05T21:07:03.144 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:03.143+0000 7f8eed5821c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-05T21:07:03.212 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:03.211+0000 7f8eed5821c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-05T21:07:03.612 INFO:tasks.ceph.osd.0.smithi070.stderr:2023-10-05T21:07:03.610+0000 7fdb43dd3540 -1 osd.0 0 log_to_monitors true 2023-10-05T21:07:03.656 INFO:tasks.ceph.osd.1.smithi070.stderr:2023-10-05T21:07:03.655+0000 7ffae20d5540 -1 osd.1 0 log_to_monitors true 2023-10-05T21:07:03.679 INFO:tasks.ceph.osd.2.smithi070.stderr:2023-10-05T21:07:03.678+0000 7f6a0ee5f540 -1 osd.2 0 log_to_monitors true 2023-10-05T21:07:03.725 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:03.724+0000 7f8eed5821c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-05T21:07:04.032 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:04.031+0000 7f8eed5821c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-05T21:07:04.170 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:04.169+0000 7f8eed5821c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-05T21:07:04.519 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:04.518+0000 7f8eed5821c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-05T21:07:04.663 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:04.662+0000 7f8eed5821c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-05T21:07:04.834 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:04.833+0000 7f8eed5821c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-05T21:07:04.944 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:04.943+0000 7f8eed5821c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-05T21:07:05.198 INFO:tasks.ceph.osd.1.smithi070.stderr:2023-10-05T21:07:05.197+0000 7ffad8cda700 -1 osd.1 0 waiting for initial osdmap 2023-10-05T21:07:05.201 INFO:tasks.ceph.osd.1.smithi070.stderr:2023-10-05T21:07:05.199+0000 7ffad1ad1700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T21:07:05.596 INFO:tasks.ceph.osd.0.smithi070.stderr:2023-10-05T21:07:05.595+0000 7fdb3a9d8700 -1 osd.0 0 waiting for initial osdmap 2023-10-05T21:07:05.597 INFO:tasks.ceph.osd.0.smithi070.stderr:2023-10-05T21:07:05.597+0000 7fdb337cf700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T21:07:05.968 INFO:tasks.ceph.osd.2.smithi070.stderr:2023-10-05T21:07:05.967+0000 7f6a05a64700 -1 osd.2 0 waiting for initial osdmap 2023-10-05T21:07:05.973 INFO:tasks.ceph.mgr.x.smithi070.stderr:2023-10-05T21:07:05.972+0000 7f8eb1c63700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-05T21:07:05.974 INFO:tasks.ceph.osd.2.smithi070.stderr:2023-10-05T21:07:05.973+0000 7f69fe85b700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T21:07:07.349 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T21:07:07.665 INFO:teuthology.misc.health.smithi070.stdout: 2023-10-05T21:07:07.665 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":9,"fsid":"18ea46c7-744d-428a-a77f-8692a242600e","created":"2023-10-05T21:06:59.300586+0000","modified":"2023-10-05T21:07:06.668282+0000","last_up_change":"2023-10-05T21:07:06.668282+0000","last_in_change":"2023-10-05T21:07:00.278515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"7481092a-4057-4980-8713-86e63cc546cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":114355},{"type":"v1","addr":"172.21.15.70:6801","nonce":114355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":114355},{"type":"v1","addr":"172.21.15.70:6803","nonce":114355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":114355},{"type":"v1","addr":"172.21.15.70:6807","nonce":114355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":114355},{"type":"v1","addr":"172.21.15.70:6805","nonce":114355}]},"public_addr":"172.21.15.70:6801/114355","cluster_addr":"172.21.15.70:6803/114355","heartbeat_back_addr":"172.21.15.70:6807/114355","heartbeat_front_addr":"172.21.15.70:6805/114355","state":["exists","up"]},{"osd":1,"uuid":"cc290433-0501-4fe0-85c0-a82ae80e3cea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":114354},{"type":"v1","addr":"172.21.15.70:6809","nonce":114354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":114354},{"type":"v1","addr":"172.21.15.70:6811","nonce":114354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":114354},{"type":"v1","addr":"172.21.15.70:6815","nonce":114354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":114354},{"type":"v1","addr":"172.21.15.70:6813","nonce":114354}]},"public_addr":"172.21.15.70:6809/114354","cluster_addr":"172.21.15.70:6811/114354","heartbeat_back_addr":"172.21.15.70:6815/114354","heartbeat_front_addr":"172.21.15.70:6813/114354","state":["exists","up"]},{"osd":2,"uuid":"cf0ef326-b20b-48ad-82c7-6ff93ca6770c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":114384},{"type":"v1","addr":"172.21.15.70:6817","nonce":114384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":114384},{"type":"v1","addr":"172.21.15.70:6819","nonce":114384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":114384},{"type":"v1","addr":"172.21.15.70:6823","nonce":114384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":114384},{"type":"v1","addr":"172.21.15.70:6821","nonce":114384}]},"public_addr":"172.21.15.70:6817/114384","cluster_addr":"172.21.15.70:6819/114384","heartbeat_back_addr":"172.21.15.70:6823/114384","heartbeat_front_addr":"172.21.15.70:6821/114384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T21:07:04.586642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T21:07:05.198645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T21:07:07.677 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-05T21:07:07.678 INFO:tasks.ceph:Creating RBD pool 2023-10-05T21:07:07.678 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-05T21:07:08.685 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'rbd' created 2023-10-05T21:07:08.699 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-05T21:07:09.688 INFO:teuthology.orchestra.run.smithi070.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-05T21:07:09.700 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-05T21:07:09.700 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-05T21:07:09.700 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-05T21:07:09.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-05T21:07:09.701 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-05T21:07:10.071 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:07:10.084 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":113942},{"type":"v1","addr":"172.21.15.70:6825","nonce":113942}]},"active_addr":"172.21.15.70:6825/113942","active_change":"2023-10-05T21:07:04.956316+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3437251652}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":370026299}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":1079250660}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3559793583}]}]} 2023-10-05T21:07:10.086 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-05T21:07:10.086 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-05T21:07:10.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T21:07:10.393 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:07:10.393 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":12,"fsid":"18ea46c7-744d-428a-a77f-8692a242600e","created":"2023-10-05T21:06:59.300586+0000","modified":"2023-10-05T21:07:09.677306+0000","last_up_change":"2023-10-05T21:07:06.668282+0000","last_in_change":"2023-10-05T21:07:00.278515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T21:07:07.011067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T21:07:07.999964+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7481092a-4057-4980-8713-86e63cc546cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":114355},{"type":"v1","addr":"172.21.15.70:6801","nonce":114355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":114355},{"type":"v1","addr":"172.21.15.70:6803","nonce":114355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":114355},{"type":"v1","addr":"172.21.15.70:6807","nonce":114355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":114355},{"type":"v1","addr":"172.21.15.70:6805","nonce":114355}]},"public_addr":"172.21.15.70:6801/114355","cluster_addr":"172.21.15.70:6803/114355","heartbeat_back_addr":"172.21.15.70:6807/114355","heartbeat_front_addr":"172.21.15.70:6805/114355","state":["exists","up"]},{"osd":1,"uuid":"cc290433-0501-4fe0-85c0-a82ae80e3cea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":114354},{"type":"v1","addr":"172.21.15.70:6809","nonce":114354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":114354},{"type":"v1","addr":"172.21.15.70:6811","nonce":114354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":114354},{"type":"v1","addr":"172.21.15.70:6815","nonce":114354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":114354},{"type":"v1","addr":"172.21.15.70:6813","nonce":114354}]},"public_addr":"172.21.15.70:6809/114354","cluster_addr":"172.21.15.70:6811/114354","heartbeat_back_addr":"172.21.15.70:6815/114354","heartbeat_front_addr":"172.21.15.70:6813/114354","state":["exists","up"]},{"osd":2,"uuid":"cf0ef326-b20b-48ad-82c7-6ff93ca6770c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":114384},{"type":"v1","addr":"172.21.15.70:6817","nonce":114384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":114384},{"type":"v1","addr":"172.21.15.70:6819","nonce":114384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":114384},{"type":"v1","addr":"172.21.15.70:6823","nonce":114384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":114384},{"type":"v1","addr":"172.21.15.70:6821","nonce":114384}]},"public_addr":"172.21.15.70:6817/114384","cluster_addr":"172.21.15.70:6819/114384","heartbeat_back_addr":"172.21.15.70:6823/114384","heartbeat_front_addr":"172.21.15.70:6821/114384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T21:07:04.586642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T21:07:05.198645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T21:07:04.673759+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T21:07:10.406 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-05T21:07:10.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T21:07:10.717 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:07:10.717 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":13,"fsid":"18ea46c7-744d-428a-a77f-8692a242600e","created":"2023-10-05T21:06:59.300586+0000","modified":"2023-10-05T21:07:10.680476+0000","last_up_change":"2023-10-05T21:07:06.668282+0000","last_in_change":"2023-10-05T21:07:00.278515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T21:07:07.011067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T21:07:07.999964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7481092a-4057-4980-8713-86e63cc546cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":114355},{"type":"v1","addr":"172.21.15.70:6801","nonce":114355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":114355},{"type":"v1","addr":"172.21.15.70:6803","nonce":114355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":114355},{"type":"v1","addr":"172.21.15.70:6807","nonce":114355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":114355},{"type":"v1","addr":"172.21.15.70:6805","nonce":114355}]},"public_addr":"172.21.15.70:6801/114355","cluster_addr":"172.21.15.70:6803/114355","heartbeat_back_addr":"172.21.15.70:6807/114355","heartbeat_front_addr":"172.21.15.70:6805/114355","state":["exists","up"]},{"osd":1,"uuid":"cc290433-0501-4fe0-85c0-a82ae80e3cea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":114354},{"type":"v1","addr":"172.21.15.70:6809","nonce":114354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":114354},{"type":"v1","addr":"172.21.15.70:6811","nonce":114354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":114354},{"type":"v1","addr":"172.21.15.70:6815","nonce":114354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":114354},{"type":"v1","addr":"172.21.15.70:6813","nonce":114354}]},"public_addr":"172.21.15.70:6809/114354","cluster_addr":"172.21.15.70:6811/114354","heartbeat_back_addr":"172.21.15.70:6815/114354","heartbeat_front_addr":"172.21.15.70:6813/114354","state":["exists","up"]},{"osd":2,"uuid":"cf0ef326-b20b-48ad-82c7-6ff93ca6770c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":114384},{"type":"v1","addr":"172.21.15.70:6817","nonce":114384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":114384},{"type":"v1","addr":"172.21.15.70:6819","nonce":114384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":114384},{"type":"v1","addr":"172.21.15.70:6823","nonce":114384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":114384},{"type":"v1","addr":"172.21.15.70:6821","nonce":114384}]},"public_addr":"172.21.15.70:6817/114384","cluster_addr":"172.21.15.70:6819/114384","heartbeat_back_addr":"172.21.15.70:6823/114384","heartbeat_front_addr":"172.21.15.70:6821/114384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T21:07:04.586642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T21:07:05.198645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T21:07:04.673759+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T21:07:10.730 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-05T21:07:10.733 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-05T21:07:10.733 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-05T21:07:10.891 INFO:teuthology.orchestra.run.smithi070.stdout:34359738370 2023-10-05T21:07:10.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-05T21:07:10.901 INFO:teuthology.orchestra.run.smithi070.stdout:38654705666 2023-10-05T21:07:10.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-05T21:07:10.902 INFO:teuthology.orchestra.run.smithi070.stdout:34359738370 2023-10-05T21:07:10.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-05T21:07:11.219 INFO:teuthology.orchestra.run.smithi070.stdout:34359738369 2023-10-05T21:07:11.233 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.1 2023-10-05T21:07:11.267 INFO:teuthology.orchestra.run.smithi070.stdout:34359738369 2023-10-05T21:07:11.273 INFO:teuthology.orchestra.run.smithi070.stdout:38654705665 2023-10-05T21:07:11.281 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2023-10-05T21:07:11.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.2 2023-10-05T21:07:12.234 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-05T21:07:12.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-05T21:07:12.288 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-05T21:07:12.572 INFO:teuthology.orchestra.run.smithi070.stdout:34359738370 2023-10-05T21:07:12.586 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.1 2023-10-05T21:07:12.586 DEBUG:teuthology.parallel:result is None 2023-10-05T21:07:12.618 INFO:teuthology.orchestra.run.smithi070.stdout:34359738370 2023-10-05T21:07:12.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.0 2023-10-05T21:07:12.632 DEBUG:teuthology.parallel:result is None 2023-10-05T21:07:12.658 INFO:teuthology.orchestra.run.smithi070.stdout:38654705666 2023-10-05T21:07:12.672 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.2 2023-10-05T21:07:12.672 DEBUG:teuthology.parallel:result is None 2023-10-05T21:07:12.672 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-05T21:07:12.672 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-05T21:07:13.016 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:07:13.019 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2023-10-05T21:07:13.029 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-10-05T21:07:12.970887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":893600,"kb_used_data":1480,"kb_used_omap":0,"kb_used_meta":891840,"kb_avail":313679200,"statfs":{"total":322122547200,"available":321207500800,"internally_reserved":0,"allocated":1515520,"data_stored":1269063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913244160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000283"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.687266+0000","last_change":"2023-10-05T21:07:09.686550+0000","last_active":"2023-10-05T21:07:09.687266+0000","last_peered":"2023-10-05T21:07:09.687266+0000","last_clean":"2023-10-05T21:07:09.687266+0000","last_became_active":"2023-10-05T21:07:09.686245+0000","last_became_peered":"2023-10-05T21:07:09.686245+0000","last_unstale":"2023-10-05T21:07:09.687266+0000","last_undegraded":"2023-10-05T21:07:09.687266+0000","last_fullsized":"2023-10-05T21:07:09.687266+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:57:47.737840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.685787+0000","last_change":"2023-10-05T21:07:09.684027+0000","last_active":"2023-10-05T21:07:09.685787+0000","last_peered":"2023-10-05T21:07:09.685787+0000","last_clean":"2023-10-05T21:07:09.685787+0000","last_became_active":"2023-10-05T21:07:09.683739+0000","last_became_peered":"2023-10-05T21:07:09.683739+0000","last_unstale":"2023-10-05T21:07:09.685787+0000","last_undegraded":"2023-10-05T21:07:09.685787+0000","last_fullsized":"2023-10-05T21:07:09.685787+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:28:24.632356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.687278+0000","last_change":"2023-10-05T21:07:09.686579+0000","last_active":"2023-10-05T21:07:09.687278+0000","last_peered":"2023-10-05T21:07:09.687278+0000","last_clean":"2023-10-05T21:07:09.687278+0000","last_became_active":"2023-10-05T21:07:09.686311+0000","last_became_peered":"2023-10-05T21:07:09.686311+0000","last_unstale":"2023-10-05T21:07:09.687278+0000","last_undegraded":"2023-10-05T21:07:09.687278+0000","last_fullsized":"2023-10-05T21:07:09.687278+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:00:15.792161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.685618+0000","last_change":"2023-10-05T21:07:09.685618+0000","last_active":"2023-10-05T21:07:09.685618+0000","last_peered":"2023-10-05T21:07:09.685618+0000","last_clean":"2023-10-05T21:07:09.685618+0000","last_became_active":"2023-10-05T21:07:09.685375+0000","last_became_peered":"2023-10-05T21:07:09.685375+0000","last_unstale":"2023-10-05T21:07:09.685618+0000","last_undegraded":"2023-10-05T21:07:09.685618+0000","last_fullsized":"2023-10-05T21:07:09.685618+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T00:05:23.869889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":137,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.682693+0000","last_change":"2023-10-05T21:07:08.684289+0000","last_active":"2023-10-05T21:07:09.682693+0000","last_peered":"2023-10-05T21:07:09.682693+0000","last_clean":"2023-10-05T21:07:09.682693+0000","last_became_active":"2023-10-05T21:07:08.683525+0000","last_became_peered":"2023-10-05T21:07:08.683525+0000","last_unstale":"2023-10-05T21:07:09.682693+0000","last_undegraded":"2023-10-05T21:07:09.682693+0000","last_fullsized":"2023-10-05T21:07:09.682693+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:07.671600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:07.671600+0000","last_clean_scrub_stamp":"2023-10-05T21:07:07.671600+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:10:03.025123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.685771+0000","last_change":"2023-10-05T21:07:09.684166+0000","last_active":"2023-10-05T21:07:09.685771+0000","last_peered":"2023-10-05T21:07:09.685771+0000","last_clean":"2023-10-05T21:07:09.685771+0000","last_became_active":"2023-10-05T21:07:09.683892+0000","last_became_peered":"2023-10-05T21:07:09.683892+0000","last_unstale":"2023-10-05T21:07:09.685771+0000","last_undegraded":"2023-10-05T21:07:09.685771+0000","last_fullsized":"2023-10-05T21:07:09.685771+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:51:13.213122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.684218+0000","last_change":"2023-10-05T21:07:09.684218+0000","last_active":"2023-10-05T21:07:09.684218+0000","last_peered":"2023-10-05T21:07:09.684218+0000","last_clean":"2023-10-05T21:07:09.684218+0000","last_became_active":"2023-10-05T21:07:09.683937+0000","last_became_peered":"2023-10-05T21:07:09.683937+0000","last_unstale":"2023-10-05T21:07:09.684218+0000","last_undegraded":"2023-10-05T21:07:09.684218+0000","last_fullsized":"2023-10-05T21:07:09.684218+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:48:56.682323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.684163+0000","last_change":"2023-10-05T21:07:09.684163+0000","last_active":"2023-10-05T21:07:09.684163+0000","last_peered":"2023-10-05T21:07:09.684163+0000","last_clean":"2023-10-05T21:07:09.684163+0000","last_became_active":"2023-10-05T21:07:09.683902+0000","last_became_peered":"2023-10-05T21:07:09.683902+0000","last_unstale":"2023-10-05T21:07:09.684163+0000","last_undegraded":"2023-10-05T21:07:09.684163+0000","last_fullsized":"2023-10-05T21:07:09.684163+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:10:04.027808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.687298+0000","last_change":"2023-10-05T21:07:09.686739+0000","last_active":"2023-10-05T21:07:09.687298+0000","last_peered":"2023-10-05T21:07:09.687298+0000","last_clean":"2023-10-05T21:07:09.687298+0000","last_became_active":"2023-10-05T21:07:09.686313+0000","last_became_peered":"2023-10-05T21:07:09.686313+0000","last_unstale":"2023-10-05T21:07:09.687298+0000","last_undegraded":"2023-10-05T21:07:09.687298+0000","last_fullsized":"2023-10-05T21:07:09.687298+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:45:15.247599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297392,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560208,"statfs":{"total":107374182400,"available":107069652992,"internally_reserved":0,"allocated":106496,"data_stored":28287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298100,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559500,"statfs":{"total":107374182400,"available":107068928000,"internally_reserved":0,"allocated":700416,"data_stored":618655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298108,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559492,"statfs":{"total":107374182400,"available":107068919808,"internally_reserved":0,"allocated":708608,"data_stored":622121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-05T21:07:13.031 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-05T21:07:13.333 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:07:13.335 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2023-10-05T21:07:13.346 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-10-05T21:07:12.970887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":893600,"kb_used_data":1480,"kb_used_omap":0,"kb_used_meta":891840,"kb_avail":313679200,"statfs":{"total":322122547200,"available":321207500800,"internally_reserved":0,"allocated":1515520,"data_stored":1269063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913244160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000283"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.687266+0000","last_change":"2023-10-05T21:07:09.686550+0000","last_active":"2023-10-05T21:07:09.687266+0000","last_peered":"2023-10-05T21:07:09.687266+0000","last_clean":"2023-10-05T21:07:09.687266+0000","last_became_active":"2023-10-05T21:07:09.686245+0000","last_became_peered":"2023-10-05T21:07:09.686245+0000","last_unstale":"2023-10-05T21:07:09.687266+0000","last_undegraded":"2023-10-05T21:07:09.687266+0000","last_fullsized":"2023-10-05T21:07:09.687266+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:57:47.737840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.685787+0000","last_change":"2023-10-05T21:07:09.684027+0000","last_active":"2023-10-05T21:07:09.685787+0000","last_peered":"2023-10-05T21:07:09.685787+0000","last_clean":"2023-10-05T21:07:09.685787+0000","last_became_active":"2023-10-05T21:07:09.683739+0000","last_became_peered":"2023-10-05T21:07:09.683739+0000","last_unstale":"2023-10-05T21:07:09.685787+0000","last_undegraded":"2023-10-05T21:07:09.685787+0000","last_fullsized":"2023-10-05T21:07:09.685787+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:28:24.632356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.687278+0000","last_change":"2023-10-05T21:07:09.686579+0000","last_active":"2023-10-05T21:07:09.687278+0000","last_peered":"2023-10-05T21:07:09.687278+0000","last_clean":"2023-10-05T21:07:09.687278+0000","last_became_active":"2023-10-05T21:07:09.686311+0000","last_became_peered":"2023-10-05T21:07:09.686311+0000","last_unstale":"2023-10-05T21:07:09.687278+0000","last_undegraded":"2023-10-05T21:07:09.687278+0000","last_fullsized":"2023-10-05T21:07:09.687278+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:00:15.792161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.685618+0000","last_change":"2023-10-05T21:07:09.685618+0000","last_active":"2023-10-05T21:07:09.685618+0000","last_peered":"2023-10-05T21:07:09.685618+0000","last_clean":"2023-10-05T21:07:09.685618+0000","last_became_active":"2023-10-05T21:07:09.685375+0000","last_became_peered":"2023-10-05T21:07:09.685375+0000","last_unstale":"2023-10-05T21:07:09.685618+0000","last_undegraded":"2023-10-05T21:07:09.685618+0000","last_fullsized":"2023-10-05T21:07:09.685618+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T00:05:23.869889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":137,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.682693+0000","last_change":"2023-10-05T21:07:08.684289+0000","last_active":"2023-10-05T21:07:09.682693+0000","last_peered":"2023-10-05T21:07:09.682693+0000","last_clean":"2023-10-05T21:07:09.682693+0000","last_became_active":"2023-10-05T21:07:08.683525+0000","last_became_peered":"2023-10-05T21:07:08.683525+0000","last_unstale":"2023-10-05T21:07:09.682693+0000","last_undegraded":"2023-10-05T21:07:09.682693+0000","last_fullsized":"2023-10-05T21:07:09.682693+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:07.671600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:07.671600+0000","last_clean_scrub_stamp":"2023-10-05T21:07:07.671600+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:10:03.025123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.685771+0000","last_change":"2023-10-05T21:07:09.684166+0000","last_active":"2023-10-05T21:07:09.685771+0000","last_peered":"2023-10-05T21:07:09.685771+0000","last_clean":"2023-10-05T21:07:09.685771+0000","last_became_active":"2023-10-05T21:07:09.683892+0000","last_became_peered":"2023-10-05T21:07:09.683892+0000","last_unstale":"2023-10-05T21:07:09.685771+0000","last_undegraded":"2023-10-05T21:07:09.685771+0000","last_fullsized":"2023-10-05T21:07:09.685771+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:51:13.213122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.684218+0000","last_change":"2023-10-05T21:07:09.684218+0000","last_active":"2023-10-05T21:07:09.684218+0000","last_peered":"2023-10-05T21:07:09.684218+0000","last_clean":"2023-10-05T21:07:09.684218+0000","last_became_active":"2023-10-05T21:07:09.683937+0000","last_became_peered":"2023-10-05T21:07:09.683937+0000","last_unstale":"2023-10-05T21:07:09.684218+0000","last_undegraded":"2023-10-05T21:07:09.684218+0000","last_fullsized":"2023-10-05T21:07:09.684218+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:48:56.682323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.684163+0000","last_change":"2023-10-05T21:07:09.684163+0000","last_active":"2023-10-05T21:07:09.684163+0000","last_peered":"2023-10-05T21:07:09.684163+0000","last_clean":"2023-10-05T21:07:09.684163+0000","last_became_active":"2023-10-05T21:07:09.683902+0000","last_became_peered":"2023-10-05T21:07:09.683902+0000","last_unstale":"2023-10-05T21:07:09.684163+0000","last_undegraded":"2023-10-05T21:07:09.684163+0000","last_fullsized":"2023-10-05T21:07:09.684163+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:10:04.027808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-05T21:07:09.687298+0000","last_change":"2023-10-05T21:07:09.686739+0000","last_active":"2023-10-05T21:07:09.687298+0000","last_peered":"2023-10-05T21:07:09.687298+0000","last_clean":"2023-10-05T21:07:09.687298+0000","last_became_active":"2023-10-05T21:07:09.686313+0000","last_became_peered":"2023-10-05T21:07:09.686313+0000","last_unstale":"2023-10-05T21:07:09.687298+0000","last_undegraded":"2023-10-05T21:07:09.687298+0000","last_fullsized":"2023-10-05T21:07:09.687298+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T21:07:08.674285+0000","last_clean_scrub_stamp":"2023-10-05T21:07:08.674285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:45:15.247599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297392,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560208,"statfs":{"total":107374182400,"available":107069652992,"internally_reserved":0,"allocated":106496,"data_stored":28287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298100,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559500,"statfs":{"total":107374182400,"available":107068928000,"internally_reserved":0,"allocated":700416,"data_stored":618655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298108,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559492,"statfs":{"total":107374182400,"available":107068919808,"internally_reserved":0,"allocated":708608,"data_stored":622121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-05T21:07:13.347 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-05T21:07:13.347 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-05T21:07:13.347 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-05T21:07:13.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-05T21:07:13.690 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:07:13.691 INFO:teuthology.orchestra.run.smithi070.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-05T21:07:13.704 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-05T21:07:13.704 INFO:teuthology.run_tasks:Running task vault... 2023-10-05T21:07:13.746 DEBUG:tasks.vault:Vault config is {'client.0': {'engine': 'transit', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/transit/keys/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1'}, {'path': 'my-key-2'}]}} 2023-10-05T21:07:13.747 INFO:tasks.vault:Downloading Vault... 2023-10-05T21:07:13.747 INFO:tasks.vault:Downloading Vault... 2023-10-05T21:07:13.747 DEBUG:teuthology.orchestra.run.smithi070:> curl -L https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip -o /home/ubuntu/cephtest/vault.zip 2023-10-05T21:07:13.774 INFO:teuthology.orchestra.run.smithi070.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T21:07:13.774 INFO:teuthology.orchestra.run.smithi070.stderr: Dload Upload Total Spent Left Speed 2023-10-05T21:07:14.275 INFO:teuthology.orchestra.run.smithi070.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 42.6M 100 42.6M 0 0 85.0M 0 --:--:-- --:--:-- --:--:-- 85.0M 2023-10-05T21:07:14.277 INFO:tasks.vault:Verifying SHA256 signature... 2023-10-05T21:07:14.277 DEBUG:teuthology.orchestra.run.smithi070:> echo '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 /home/ubuntu/cephtest/vault.zip' | sha256sum --check --status 2023-10-05T21:07:14.454 INFO:tasks.vault:Extracting vault... 2023-10-05T21:07:14.454 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p /home/ubuntu/cephtest/vault 2023-10-05T21:07:14.472 DEBUG:teuthology.orchestra.run.smithi070:> unzip /home/ubuntu/cephtest/vault.zip -d /home/ubuntu/cephtest/vault 2023-10-05T21:07:14.539 INFO:teuthology.orchestra.run.smithi070.stdout:Archive: /home/ubuntu/cephtest/vault.zip 2023-10-05T21:07:15.263 INFO:teuthology.orchestra.run.smithi070.stdout: inflating: /home/ubuntu/cephtest/vault/vault 2023-10-05T21:07:15.263 INFO:tasks.vault:Starting Vault listening on 0.0.0.0:8200 ... 2023-10-05T21:07:15.263 INFO:tasks.vault.client.0:Restarting daemon 2023-10-05T21:07:15.263 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/vault && exec bash -c 'chmod +x /home/ubuntu/cephtest/vault/vault && /home/ubuntu/cephtest/vault/vault server -dev -dev-listen-address=0.0.0.0:8200 -dev-no-store-token -dev-root-token-id=test_root_token' & { read; kill %1; }) 2023-10-05T21:07:15.305 INFO:tasks.vault.client.0:Started 2023-10-05T21:07:15.351 INFO:tasks.vault.client.0.smithi070.stdout:==> Vault server configuration: 2023-10-05T21:07:15.351 INFO:tasks.vault.client.0.smithi070.stdout: 2023-10-05T21:07:15.351 INFO:tasks.vault.client.0.smithi070.stdout: Api Address: http://0.0.0.0:8200 2023-10-05T21:07:15.351 INFO:tasks.vault.client.0.smithi070.stdout: Cgo: disabled 2023-10-05T21:07:15.351 INFO:tasks.vault.client.0.smithi070.stdout: Cluster Address: https://0.0.0.0:8201 2023-10-05T21:07:15.351 INFO:tasks.vault.client.0.smithi070.stdout: Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled") 2023-10-05T21:07:15.351 INFO:tasks.vault.client.0.smithi070.stdout: Log Level: info 2023-10-05T21:07:15.351 INFO:tasks.vault.client.0.smithi070.stdout: Mlock: supported: true, enabled: false 2023-10-05T21:07:15.351 INFO:tasks.vault.client.0.smithi070.stdout: Storage: inmem 2023-10-05T21:07:15.352 INFO:tasks.vault.client.0.smithi070.stdout: Version: Vault v1.2.2 2023-10-05T21:07:15.352 INFO:tasks.vault.client.0.smithi070.stdout: 2023-10-05T21:07:15.369 INFO:tasks.vault.client.0.smithi070.stdout:WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory 2023-10-05T21:07:15.369 INFO:tasks.vault.client.0.smithi070.stdout:and starts unsealed with a single unseal key. The root token is already 2023-10-05T21:07:15.369 INFO:tasks.vault.client.0.smithi070.stdout:authenticated to the CLI, so you can immediately begin using Vault. 2023-10-05T21:07:15.369 INFO:tasks.vault.client.0.smithi070.stdout: 2023-10-05T21:07:15.369 INFO:tasks.vault.client.0.smithi070.stdout:You may need to set the following environment variable: 2023-10-05T21:07:15.370 INFO:tasks.vault.client.0.smithi070.stdout: 2023-10-05T21:07:15.370 INFO:tasks.vault.client.0.smithi070.stdout: $ export VAULT_ADDR='http://0.0.0.0:8200' 2023-10-05T21:07:15.370 INFO:tasks.vault.client.0.smithi070.stdout: 2023-10-05T21:07:15.370 INFO:tasks.vault.client.0.smithi070.stdout:The unseal key and root token are displayed below in case you want to 2023-10-05T21:07:15.370 INFO:tasks.vault.client.0.smithi070.stdout:seal/unseal the Vault or re-authenticate. 2023-10-05T21:07:15.370 INFO:tasks.vault.client.0.smithi070.stdout: 2023-10-05T21:07:15.370 INFO:tasks.vault.client.0.smithi070.stdout:Unseal Key: petuFhktmfFEMn+4HwUb2/oRkJ6NOVPPAs4G4EPIV7M= 2023-10-05T21:07:15.370 INFO:tasks.vault.client.0.smithi070.stdout:Root Token: test_root_token 2023-10-05T21:07:15.370 INFO:tasks.vault.client.0.smithi070.stdout: 2023-10-05T21:07:15.370 INFO:tasks.vault.client.0.smithi070.stdout:Development mode should NOT be used in production installations! 2023-10-05T21:07:15.370 INFO:tasks.vault.client.0.smithi070.stdout: 2023-10-05T21:07:15.370 INFO:tasks.vault.client.0.smithi070.stdout:==> Vault server started! Log data will stream in below: 2023-10-05T21:07:15.371 INFO:tasks.vault.client.0.smithi070.stdout: 2023-10-05T21:07:15.371 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.349Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set 2023-10-05T21:07:15.371 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.349Z [INFO] core: security barrier not initialized 2023-10-05T21:07:15.371 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.349Z [INFO] core: security barrier initialized: shares=1 threshold=1 2023-10-05T21:07:15.371 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.350Z [INFO] core: post-unseal setup starting 2023-10-05T21:07:15.371 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.357Z [INFO] core: loaded wrapping token key 2023-10-05T21:07:15.371 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.357Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.357Z [INFO] core: no mounts; adding default mount table 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.358Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.358Z [INFO] core: successfully mounted backend: type=system path=sys/ 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.359Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.359Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.359Z [INFO] core: restoring leases 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.359Z [INFO] rollback: starting rollback manager 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.360Z [INFO] expiration: lease restore complete 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.360Z [INFO] identity: entities restored 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.360Z [INFO] identity: groups restored 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.360Z [INFO] core: post-unseal setup complete 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.361Z [INFO] core: root token generated 2023-10-05T21:07:15.372 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.361Z [INFO] core: pre-seal teardown starting 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.361Z [INFO] rollback: stopping rollback manager 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.361Z [INFO] core: pre-seal teardown complete 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.361Z [INFO] core.cluster-listener: starting listener: listener_address=0.0.0.0:8201 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.361Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.361Z [INFO] core: post-unseal setup starting 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.361Z [INFO] core: loaded wrapping token key 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.361Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.361Z [INFO] core: successfully mounted backend: type=system path=sys/ 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.361Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.361Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.362Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.362Z [INFO] core: restoring leases 2023-10-05T21:07:15.373 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.362Z [INFO] rollback: starting rollback manager 2023-10-05T21:07:15.374 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.362Z [INFO] identity: entities restored 2023-10-05T21:07:15.374 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.362Z [INFO] expiration: lease restore complete 2023-10-05T21:07:15.374 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.362Z [INFO] identity: groups restored 2023-10-05T21:07:15.374 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.362Z [INFO] core: post-unseal setup complete 2023-10-05T21:07:15.374 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.362Z [INFO] core: vault is unsealed 2023-10-05T21:07:15.374 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.363Z [INFO] expiration: revoked lease: lease_id=auth/token/root/h90e823b4bab0710fa93e610b9cd27e37105aab5fe3af13da51b4eaa5ff870ad5 2023-10-05T21:07:15.374 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.364Z [INFO] core: successful mount: namespace= path=secret/ type=kv 2023-10-05T21:07:15.375 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.374Z [INFO] secrets.kv.kv_a48889d5: collecting keys to upgrade 2023-10-05T21:07:15.375 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.374Z [INFO] secrets.kv.kv_a48889d5: done collecting keys: num_keys=1 2023-10-05T21:07:15.375 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:15.374Z [INFO] secrets.kv.kv_a48889d5: upgrading keys finished 2023-10-05T21:07:25.306 INFO:tasks.vault:Mounting transit secrets engine 2023-10-05T21:07:25.306 INFO:tasks.vault:Send request to Vault: smithi070.front.sepia.ceph.com:8200 at /v1/sys/mounts/transit with token: test_root_token 2023-10-05T21:07:25.313 INFO:tasks.vault:b'' 2023-10-05T21:07:25.313 INFO:tasks.vault:Send request to Vault: smithi070.front.sepia.ceph.com:8200 at /v1/transit/keys/my-key-1 with token: test_root_token 2023-10-05T21:07:25.314 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:07:25.312Z [INFO] core: successful mount: namespace= path=transit/ type=transit 2023-10-05T21:07:25.315 INFO:tasks.vault:b'' 2023-10-05T21:07:25.315 INFO:tasks.vault:Send request to Vault: smithi070.front.sepia.ceph.com:8200 at /v1/transit/keys/my-key-2 with token: test_root_token 2023-10-05T21:07:25.318 INFO:tasks.vault:b'' 2023-10-05T21:07:25.318 INFO:tasks.vault:secrets created 2023-10-05T21:07:25.318 INFO:teuthology.run_tasks:Running task rgw... 2023-10-05T21:07:25.331 DEBUG:tasks.rgw:config is {'client.0': {'use-vault-role': 'client.0'}} 2023-10-05T21:07:25.331 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-10-05T21:07:25.332 INFO:tasks.rgw:Creating data pools 2023-10-05T21:07:25.332 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-10-05T21:07:25.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2023-10-05T21:07:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'default.rgw.buckets.data' created 2023-10-05T21:07:26.315 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-10-05T21:07:27.313 INFO:teuthology.orchestra.run.smithi070.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-05T21:07:27.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-10-05T21:07:28.308 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'default.rgw.buckets.index' created 2023-10-05T21:07:28.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-10-05T21:07:29.339 INFO:teuthology.orchestra.run.smithi070.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-05T21:07:29.350 DEBUG:tasks.rgw:Pools created 2023-10-05T21:07:29.350 INFO:tasks.rgw:Starting rgw... 2023-10-05T21:07:29.350 INFO:tasks.rgw:rgw client.0 config is {'use-vault-role': 'client.0'} 2023-10-05T21:07:29.350 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-05T21:07:29.351 DEBUG:teuthology.orchestra.run.smithi070:> sudo echo -n test_root_token | sudo tee /etc/ceph/vault-root-token 2023-10-05T21:07:29.379 INFO:teuthology.orchestra.run.smithi070.stdout:test_root_token 2023-10-05T21:07:29.380 INFO:tasks.rgw:Token file content 2023-10-05T21:07:29.380 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/ceph/vault-root-token 2023-10-05T21:07:29.437 INFO:teuthology.orchestra.run.smithi070.stdout:test_root_token 2023-10-05T21:07:29.438 INFO:tasks.rgw:Restrict access to token file 2023-10-05T21:07:29.438 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 600 /etc/ceph/vault-root-token 2023-10-05T21:07:29.504 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown ceph /etc/ceph/vault-root-token 2023-10-05T21:07:29.572 INFO:tasks.rgw.client.0:Restarting daemon 2023-10-05T21:07:29.572 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi070.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --rgw_crypt_sse_s3_vault_addr smithi070.front.sepia.ceph.com:8200 --rgw_crypt_sse_s3_vault_token_file /etc/ceph/vault-root-token --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-10-05T21:07:29.614 INFO:tasks.rgw.client.0:Started 2023-10-05T21:07:29.614 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi070.front.sepia.ceph.com:80/ 2023-10-05T21:07:29.614 DEBUG:teuthology.orchestra.run.smithi070:> curl http://smithi070.front.sepia.ceph.com:80/ 2023-10-05T21:07:29.635 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T21:07:29.635 INFO:teuthology.orchestra.run.smithi070.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T21:07:29.635 INFO:teuthology.orchestra.run.smithi070.stderr: Dload Upload Total Spent Left Speed 2023-10-05T21:07:29.636 INFO:teuthology.orchestra.run.smithi070.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi070.front.sepia.ceph.com port 80: Connection refused 2023-10-05T21:07:30.637 DEBUG:teuthology.orchestra.run.smithi070:> curl http://smithi070.front.sepia.ceph.com:80/ 2023-10-05T21:07:30.663 INFO:teuthology.orchestra.run.smithi070.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T21:07:30.663 INFO:teuthology.orchestra.run.smithi070.stderr: Dload Upload Total Spent Left Speed 2023-10-05T21:07:30.664 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T21:07:30.665 INFO:teuthology.orchestra.run.smithi070.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi070.front.sepia.ceph.com port 80: Connection refused 2023-10-05T21:07:32.665 DEBUG:teuthology.orchestra.run.smithi070:> curl http://smithi070.front.sepia.ceph.com:80/ 2023-10-05T21:07:32.692 INFO:teuthology.orchestra.run.smithi070.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T21:07:32.692 INFO:teuthology.orchestra.run.smithi070.stderr: Dload Upload Total Spent Left Speed 2023-10-05T21:07:32.693 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T21:07:32.693 INFO:teuthology.orchestra.run.smithi070.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi070.front.sepia.ceph.com port 80: Connection refused 2023-10-05T21:07:36.694 DEBUG:teuthology.orchestra.run.smithi070:> curl http://smithi070.front.sepia.ceph.com:80/ 2023-10-05T21:07:36.720 INFO:teuthology.orchestra.run.smithi070.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T21:07:36.720 INFO:teuthology.orchestra.run.smithi070.stderr: Dload Upload Total Spent Left Speed 2023-10-05T21:07:36.721 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T21:07:36.721 INFO:teuthology.orchestra.run.smithi070.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi070.front.sepia.ceph.com port 80: Connection refused 2023-10-05T21:07:44.723 DEBUG:teuthology.orchestra.run.smithi070:> curl http://smithi070.front.sepia.ceph.com:80/ 2023-10-05T21:07:44.749 INFO:teuthology.orchestra.run.smithi070.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T21:07:44.749 INFO:teuthology.orchestra.run.smithi070.stderr: Dload Upload Total Spent Left Speed 2023-10-05T21:07:44.750 INFO:teuthology.orchestra.run.smithi070.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-05T21:07:44.751 INFO:teuthology.orchestra.run.smithi070.stdout:anonymous 2023-10-05T21:07:44.751 INFO:teuthology.run_tasks:Running task s3tests... 2023-10-05T21:07:44.774 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'with-sse-s3': True}} 2023-10-05T21:07:44.774 INFO:tasks.s3tests:Downloading s3-tests... 2023-10-05T21:07:44.774 INFO:tasks.s3tests:Using branch 'ceph-quincy' for s3tests 2023-10-05T21:07:44.775 DEBUG:teuthology.orchestra.run.smithi070:> git clone -b ceph-quincy https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T21:07:44.812 INFO:teuthology.orchestra.run.smithi070.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests-client.0'... 2023-10-05T21:07:45.773 INFO:tasks.s3tests:Creating rgw users... 2023-10-05T21:07:45.774 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2023-10-05T21:07:45.774 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key DNEBPVGXKFWAQJAQIZSN --secret eYEnfFuqgiF9hxKJWNqM7X8G8X8ypbInANNLdTFeo86DSEm+KE0sbA== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-10-05T21:07:45.865 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setuser ceph since I am not root 2023-10-05T21:07:45.866 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T21:07:45.871 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.870+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.873 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.871+0000 7f19c745e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.873 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.871+0000 7f19c745e680 20 realm 2023-10-05T21:07:45.873 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.871+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.873 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.872+0000 7f19c745e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.873 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.872+0000 7f19c745e680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T21:07:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.872+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.872+0000 7f19c745e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.872+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.872+0000 7f19c745e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.872+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.872+0000 7f19c745e680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:07:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.872+0000 7f19c745e680 20 searching for the correct realm 2023-10-05T21:07:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.879+0000 7f19c745e680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:07:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.879+0000 7f19c745e680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:07:45.882 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.879+0000 7f19c745e680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:07:45.882 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.879+0000 7f19c745e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:07:45.882 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.879+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.882 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.880+0000 7f19c745e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.882 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.880+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.882 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.880+0000 7f19c745e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:45.882 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.880+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.882 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.880+0000 7f19c745e680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:07:45.882 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.880+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.882 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.880+0000 7f19c745e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.882 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.880+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.880+0000 7f19c745e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.880+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.881+0000 7f19c745e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:45.887 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.886+0000 7f19c745e680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:07:45.887 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.886+0000 7f19c745e680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:07:45.887 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.886+0000 7f19c745e680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:07:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.886+0000 7f19c745e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:07:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.886+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.889 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:07:45.889 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:07:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 20 zone default found 2023-10-05T21:07:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 4 Realm: () 2023-10-05T21:07:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 4 ZoneGroup: default (cb9d3d40-68a5-4791-bf47-2aada3a09714) 2023-10-05T21:07:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 4 Zone: default (bb6df8cb-740d-4b21-924c-79673a5d256e) 2023-10-05T21:07:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T21:07:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 20 zonegroup default 2023-10-05T21:07:45.891 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.887+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.891 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.888+0000 7f19c745e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.891 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.888+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.891 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.888+0000 7f19c745e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.891 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.888+0000 7f19c745e680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.891 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.888+0000 7f19c745e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.891 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.888+0000 7f19c745e680 20 started sync module instance, tier type = 2023-10-05T21:07:45.891 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.888+0000 7f19c745e680 20 started zone id=bb6df8cb-740d-4b21-924c-79673a5d256e (name=default) with tier type = 2023-10-05T21:07:45.897 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.896+0000 7f19c745e680 20 add_watcher() i=0 2023-10-05T21:07:45.897 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.896+0000 7f19c745e680 20 add_watcher() i=1 2023-10-05T21:07:45.897 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.896+0000 7f19c745e680 20 add_watcher() i=2 2023-10-05T21:07:45.897 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.896+0000 7f19c745e680 20 add_watcher() i=3 2023-10-05T21:07:45.897 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.896+0000 7f19c745e680 20 add_watcher() i=4 2023-10-05T21:07:45.897 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.896+0000 7f19c745e680 20 add_watcher() i=5 2023-10-05T21:07:45.897 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.896+0000 7f19c745e680 20 add_watcher() i=6 2023-10-05T21:07:45.897 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.896+0000 7f19c745e680 20 add_watcher() i=7 2023-10-05T21:07:45.897 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.896+0000 7f19c745e680 2 all 8 watchers are set, enabling cache 2023-10-05T21:07:45.899 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.898+0000 7f198bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:07:45.899 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.898+0000 7f19c745e680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T21:07:45.899 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.898+0000 7f19c745e680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T21:07:45.903 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.902+0000 7f19c745e680 5 note: GC not initialized 2023-10-05T21:07:45.905 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.903+0000 7f18faffd700 20 reqs_thread_entry: start 2023-10-05T21:07:45.905 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.903+0000 7f18f8ff9700 20 reqs_thread_entry: start 2023-10-05T21:07:45.905 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.903+0000 7f19c745e680 20 init_complete bucket index max shards: 11 2023-10-05T21:07:45.905 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.903+0000 7f19c745e680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T21:07:45.906 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.903+0000 7f18eeffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 21:08:16 2023 2023-10-05T21:07:45.906 INFO:teuthology.orchestra.run.smithi070.stderr: 2023-10-05T21:07:45.906 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.903+0000 7f18ee7fc700 20 reqs_thread_entry: start 2023-10-05T21:07:45.906 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.903+0000 7f19c745e680 20 get_system_obj_state: rctx=0x7fffb6595a88 obj=default.rgw.meta:users.uid:foo.client.0 state=0x562b7712a240 s->prefetch_data=0 2023-10-05T21:07:45.906 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.903+0000 7f19c745e680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T21:07:45.906 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2023-10-05T21:07:45.906 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T21:07:45.906 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 20 get_system_obj_state: rctx=0x7fffb6595778 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x562b7712a240 s->prefetch_data=0 2023-10-05T21:07:45.906 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2023-10-05T21:07:45.906 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2023-10-05T21:07:45.906 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T21:07:45.906 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 20 get_system_obj_state: rctx=0x7fffb6595778 obj=default.rgw.meta:users.keys:DNEBPVGXKFWAQJAQIZSN state=0x562b7712a240 s->prefetch_data=0 2023-10-05T21:07:45.907 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 10 cache get: name=default.rgw.meta+users.keys+DNEBPVGXKFWAQJAQIZSN : miss 2023-10-05T21:07:45.907 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 10 cache put: name=default.rgw.meta+users.keys+DNEBPVGXKFWAQJAQIZSN info.flags=0x0 2023-10-05T21:07:45.907 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 10 adding default.rgw.meta+users.keys+DNEBPVGXKFWAQJAQIZSN to cache LRU end 2023-10-05T21:07:45.907 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 20 get_system_obj_state: rctx=0x7fffb6595008 obj=default.rgw.meta:users.keys:DNEBPVGXKFWAQJAQIZSN state=0x562b7712a240 s->prefetch_data=0 2023-10-05T21:07:45.907 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 10 cache get: name=default.rgw.meta+users.keys+DNEBPVGXKFWAQJAQIZSN : hit (negative entry) 2023-10-05T21:07:45.907 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 20 get_system_obj_state: rctx=0x7fffb65956b8 obj=default.rgw.meta:users.keys:DNEBPVGXKFWAQJAQIZSN state=0x562b7712a240 s->prefetch_data=0 2023-10-05T21:07:45.907 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.904+0000 7f19c745e680 10 cache get: name=default.rgw.meta+users.keys+DNEBPVGXKFWAQJAQIZSN : hit (negative entry) 2023-10-05T21:07:45.908 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.905+0000 7f19c745e680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T21:07:45.908 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.905+0000 7f19c745e680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T21:07:45.908 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.905+0000 7f19c745e680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T21:07:45.908 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.905+0000 7f19abfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149696 cookie 94744682889344 notifier 4210 bl.length()=640 2023-10-05T21:07:45.908 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.906+0000 7f19abfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T21:07:45.908 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.906+0000 7f19abfff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T21:07:45.909 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.907+0000 7f19c745e680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-10-05T21:07:45.909 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.907+0000 7f19c745e680 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T21:07:45.909 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.907+0000 7f19c745e680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-10-05T21:07:45.909 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.907+0000 7f19abfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149696 cookie 94744682923648 notifier 4210 bl.length()=194 2023-10-05T21:07:45.909 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.907+0000 7f19abfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-10-05T21:07:45.909 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.907+0000 7f19abfff700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T21:07:45.910 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.908+0000 7f19c745e680 10 cache put: name=default.rgw.meta+users.keys+DNEBPVGXKFWAQJAQIZSN info.flags=0x7 2023-10-05T21:07:45.910 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.908+0000 7f19c745e680 10 moving default.rgw.meta+users.keys+DNEBPVGXKFWAQJAQIZSN to cache LRU end 2023-10-05T21:07:45.910 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.908+0000 7f19c745e680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:DNEBPVGXKFWAQJAQIZSN, ofs0, ns] 2023-10-05T21:07:45.910 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.909+0000 7f19ab7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149697 cookie 94744682914096 notifier 4210 bl.length()=186 2023-10-05T21:07:45.910 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.909+0000 7f19ab7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+DNEBPVGXKFWAQJAQIZSN info.flags=0x7 2023-10-05T21:07:45.910 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.909+0000 7f19ab7fe700 10 rgw watcher librados: moving default.rgw.meta+users.keys+DNEBPVGXKFWAQJAQIZSN to cache LRU end 2023-10-05T21:07:45.910 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2023-10-05T21:07:45.911 INFO:teuthology.orchestra.run.smithi070.stdout: "user_id": "foo.client.0", 2023-10-05T21:07:45.911 INFO:teuthology.orchestra.run.smithi070.stdout: "display_name": "Mr. foo.client.0", 2023-10-05T21:07:45.911 INFO:teuthology.orchestra.run.smithi070.stdout: "email": "foo.client.0+test@test.test", 2023-10-05T21:07:45.911 INFO:teuthology.orchestra.run.smithi070.stdout: "suspended": 0, 2023-10-05T21:07:45.911 INFO:teuthology.orchestra.run.smithi070.stdout: "max_buckets": 1000, 2023-10-05T21:07:45.911 INFO:teuthology.orchestra.run.smithi070.stdout: "subusers": [], 2023-10-05T21:07:45.911 INFO:teuthology.orchestra.run.smithi070.stdout: "keys": [ 2023-10-05T21:07:45.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2023-10-05T21:07:45.911 INFO:teuthology.orchestra.run.smithi070.stdout: "user": "foo.client.0", 2023-10-05T21:07:45.911 INFO:teuthology.orchestra.run.smithi070.stdout: "access_key": "DNEBPVGXKFWAQJAQIZSN", 2023-10-05T21:07:45.911 INFO:teuthology.orchestra.run.smithi070.stdout: "secret_key": "eYEnfFuqgiF9hxKJWNqM7X8G8X8ypbInANNLdTFeo86DSEm+KE0sbA==" 2023-10-05T21:07:45.911 INFO:teuthology.orchestra.run.smithi070.stdout: } 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: "swift_keys": [], 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": [ 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "user-policy", 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: "perm": "*" 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: } 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: "op_mask": "read, write, delete", 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: "default_placement": "", 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: "default_storage_class": "", 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: "placement_tags": [], 2023-10-05T21:07:45.912 INFO:teuthology.orchestra.run.smithi070.stdout: "bucket_quota": { 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": false, 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: "check_on_raw": false, 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size": -1, 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size_kb": 0, 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: "max_objects": -1 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: "user_quota": { 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": false, 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: "check_on_raw": false, 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size": -1, 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size_kb": 0, 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: "max_objects": -1 2023-10-05T21:07:45.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2023-10-05T21:07:45.914 INFO:teuthology.orchestra.run.smithi070.stdout: "temp_url_keys": [], 2023-10-05T21:07:45.914 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "rgw", 2023-10-05T21:07:45.914 INFO:teuthology.orchestra.run.smithi070.stdout: "mfa_ids": [] 2023-10-05T21:07:45.914 INFO:teuthology.orchestra.run.smithi070.stdout:} 2023-10-05T21:07:45.914 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:07:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.910+0000 7f19c745e680 20 remove_watcher() i=0 2023-10-05T21:07:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.910+0000 7f19c745e680 2 removed watcher, disabling cache 2023-10-05T21:07:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.911+0000 7f19c745e680 20 remove_watcher() i=1 2023-10-05T21:07:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.912+0000 7f19c745e680 20 remove_watcher() i=2 2023-10-05T21:07:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.913+0000 7f19c745e680 20 remove_watcher() i=3 2023-10-05T21:07:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.913+0000 7f19c745e680 20 remove_watcher() i=4 2023-10-05T21:07:45.915 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.914+0000 7f19c745e680 20 remove_watcher() i=5 2023-10-05T21:07:45.916 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.915+0000 7f19c745e680 20 remove_watcher() i=6 2023-10-05T21:07:45.916 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.916+0000 7f19c745e680 20 remove_watcher() i=7 2023-10-05T21:07:45.917 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.916+0000 7f198bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:07:45.922 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 6687233219 --totp-seed XWF6KH5GZSHQ64Q6SG6O3OB4DT6L2IKDESES3ZXI2KP4GVVWT5ALMUXH7J6IL33V --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T21:07:45.973 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setuser ceph since I am not root 2023-10-05T21:07:45.973 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T21:07:45.979 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.978+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.979+0000 7f73424df680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.979+0000 7f73424df680 20 realm 2023-10-05T21:07:45.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.979+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.979+0000 7f73424df680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.979+0000 7f73424df680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T21:07:45.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.979+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.979+0000 7f73424df680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.979+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.980+0000 7f73424df680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:45.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.980+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.980+0000 7f73424df680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:07:45.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.980+0000 7f73424df680 20 searching for the correct realm 2023-10-05T21:07:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.987+0000 7f73424df680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:07:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.987+0000 7f73424df680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:07:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.987+0000 7f73424df680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:07:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.987+0000 7f73424df680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:07:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.987+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.987+0000 7f73424df680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.987+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.987+0000 7f73424df680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:45.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.987+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.988+0000 7f73424df680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:07:45.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.988+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.988+0000 7f73424df680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.988+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.988+0000 7f73424df680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.988+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.988+0000 7f73424df680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:45.995 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.994+0000 7f73424df680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:07:45.996 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.994+0000 7f73424df680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:07:45.996 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.994+0000 7f73424df680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:07:45.996 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.994+0000 7f73424df680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:07:45.996 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.994+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.996 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.994+0000 7f73424df680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:45.996 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.994+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.996 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:07:45.996 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.996 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:45.996 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:07:45.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 20 zone default found 2023-10-05T21:07:45.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 4 Realm: () 2023-10-05T21:07:45.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 4 ZoneGroup: default (cb9d3d40-68a5-4791-bf47-2aada3a09714) 2023-10-05T21:07:45.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 4 Zone: default (bb6df8cb-740d-4b21-924c-79673a5d256e) 2023-10-05T21:07:45.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T21:07:45.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 20 zonegroup default 2023-10-05T21:07:45.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.995+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.996+0000 7f73424df680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.996+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.996+0000 7f73424df680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.996+0000 7f73424df680 20 started sync module instance, tier type = 2023-10-05T21:07:45.999 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:45.996+0000 7f73424df680 20 started zone id=bb6df8cb-740d-4b21-924c-79673a5d256e (name=default) with tier type = 2023-10-05T21:07:46.004 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.003+0000 7f73424df680 20 add_watcher() i=0 2023-10-05T21:07:46.004 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.003+0000 7f73424df680 20 add_watcher() i=1 2023-10-05T21:07:46.004 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.003+0000 7f73424df680 20 add_watcher() i=2 2023-10-05T21:07:46.004 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.003+0000 7f73424df680 20 add_watcher() i=3 2023-10-05T21:07:46.004 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.003+0000 7f73424df680 20 add_watcher() i=4 2023-10-05T21:07:46.004 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.003+0000 7f73424df680 20 add_watcher() i=5 2023-10-05T21:07:46.005 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.003+0000 7f73424df680 20 add_watcher() i=6 2023-10-05T21:07:46.005 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.004+0000 7f73424df680 20 add_watcher() i=7 2023-10-05T21:07:46.005 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.004+0000 7f73424df680 2 all 8 watchers are set, enabling cache 2023-10-05T21:07:46.006 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.005+0000 7f73067e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:07:46.006 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.005+0000 7f73424df680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T21:07:46.006 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.005+0000 7f73424df680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T21:07:46.010 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.009+0000 7f73424df680 5 note: GC not initialized 2023-10-05T21:07:46.011 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.010+0000 7f72757fa700 20 reqs_thread_entry: start 2023-10-05T21:07:46.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.010+0000 7f726effd700 20 reqs_thread_entry: start 2023-10-05T21:07:46.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.010+0000 7f73424df680 20 init_complete bucket index max shards: 11 2023-10-05T21:07:46.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.010+0000 7f73424df680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T21:07:46.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.010+0000 7f726d7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 21:08:16 2023 2023-10-05T21:07:46.012 INFO:teuthology.orchestra.run.smithi070.stderr: 2023-10-05T21:07:46.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f726cff9700 20 reqs_thread_entry: start 2023-10-05T21:07:46.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 20 get_system_obj_state: rctx=0x7ffef6c2e638 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55a5a09f7240 s->prefetch_data=0 2023-10-05T21:07:46.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T21:07:46.013 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-10-05T21:07:46.013 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T21:07:46.013 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:07:46.013 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T21:07:46.013 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 20 Read xattr: user.rgw.idtag 2023-10-05T21:07:46.013 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T21:07:46.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 20 rados->read ofs=0 len=0 2023-10-05T21:07:46.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 20 rados_obj.operate() r=0 bl.length=455 2023-10-05T21:07:46.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-10-05T21:07:46.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T21:07:46.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:46.011+0000 7f73424df680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:07:47.374 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.371+0000 7f73424df680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T21:07:47.374 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.371+0000 7f73424df680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T21:07:47.374 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.371+0000 7f73424df680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T21:07:47.374 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.372+0000 7f73267fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084290 cookie 94169854375248 notifier 4216 bl.length()=654 2023-10-05T21:07:47.374 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.372+0000 7f73267fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T21:07:47.375 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.372+0000 7f73267fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T21:07:47.375 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.373+0000 7f73424df680 20 remove_watcher() i=0 2023-10-05T21:07:47.375 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.373+0000 7f73424df680 2 removed watcher, disabling cache 2023-10-05T21:07:47.376 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.374+0000 7f73424df680 20 remove_watcher() i=1 2023-10-05T21:07:47.377 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.375+0000 7f73424df680 20 remove_watcher() i=2 2023-10-05T21:07:47.378 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.376+0000 7f73424df680 20 remove_watcher() i=3 2023-10-05T21:07:47.379 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.377+0000 7f73424df680 20 remove_watcher() i=4 2023-10-05T21:07:47.380 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.378+0000 7f73424df680 20 remove_watcher() i=5 2023-10-05T21:07:47.381 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.379+0000 7f73424df680 20 remove_watcher() i=6 2023-10-05T21:07:47.382 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.380+0000 7f73424df680 20 remove_watcher() i=7 2023-10-05T21:07:47.382 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.381+0000 7f73067e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:07:47.388 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2023-10-05T21:07:47.389 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key XWTSCMQDUCFJOSOPVZAH --secret QmI9XsLcWS13hBTmvXb/x6pFP2ZtCO+f7ZDICMpcfAyAqijKA3/lTA== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-10-05T21:07:47.442 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setuser ceph since I am not root 2023-10-05T21:07:47.443 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T21:07:47.449 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.447+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.450 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.448+0000 7f158a598680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.450 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.448+0000 7f158a598680 20 realm 2023-10-05T21:07:47.450 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.448+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.450 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.448+0000 7f158a598680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.450 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.448+0000 7f158a598680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T21:07:47.450 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.448+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.451 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.448+0000 7f158a598680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.451 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.448+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.451 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.448+0000 7f158a598680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.452 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.448+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.452 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.449+0000 7f158a598680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:07:47.452 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.449+0000 7f158a598680 20 searching for the correct realm 2023-10-05T21:07:47.458 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.456+0000 7f158a598680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:07:47.458 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.456+0000 7f158a598680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:07:47.458 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.456+0000 7f158a598680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:07:47.458 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.456+0000 7f158a598680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:07:47.458 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.456+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.458 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.456+0000 7f158a598680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.459 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.456+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.460 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.456+0000 7f158a598680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.460 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.456+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.460 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.457+0000 7f158a598680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:07:47.460 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.457+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.460 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.457+0000 7f158a598680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.460 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.457+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.460 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.457+0000 7f158a598680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.460 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.457+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.460 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.457+0000 7f158a598680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.465 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.463+0000 7f158a598680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:07:47.465 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.463+0000 7f158a598680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:07:47.466 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.463+0000 7f158a598680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:07:47.466 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.463+0000 7f158a598680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:07:47.466 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.463+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.466 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.464+0000 7f158a598680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.466 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.464+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.467 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.464+0000 7f158a598680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:07:47.467 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.464+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.467 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.464+0000 7f158a598680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.464+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.464+0000 7f158a598680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.464+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 20 zone default found 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 4 Realm: () 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 4 ZoneGroup: default (cb9d3d40-68a5-4791-bf47-2aada3a09714) 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 4 Zone: default (bb6df8cb-740d-4b21-924c-79673a5d256e) 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 20 zonegroup default 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.469 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.469 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.469 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.469 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 20 started sync module instance, tier type = 2023-10-05T21:07:47.469 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.465+0000 7f158a598680 20 started zone id=bb6df8cb-740d-4b21-924c-79673a5d256e (name=default) with tier type = 2023-10-05T21:07:47.474 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.472+0000 7f158a598680 20 add_watcher() i=0 2023-10-05T21:07:47.474 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.472+0000 7f158a598680 20 add_watcher() i=1 2023-10-05T21:07:47.474 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.472+0000 7f158a598680 20 add_watcher() i=2 2023-10-05T21:07:47.475 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.472+0000 7f158a598680 20 add_watcher() i=3 2023-10-05T21:07:47.475 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.472+0000 7f158a598680 20 add_watcher() i=4 2023-10-05T21:07:47.475 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.472+0000 7f158a598680 20 add_watcher() i=5 2023-10-05T21:07:47.475 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.472+0000 7f158a598680 20 add_watcher() i=6 2023-10-05T21:07:47.475 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.473+0000 7f158a598680 20 add_watcher() i=7 2023-10-05T21:07:47.475 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.473+0000 7f158a598680 2 all 8 watchers are set, enabling cache 2023-10-05T21:07:47.476 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.474+0000 7f154e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:07:47.476 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.474+0000 7f158a598680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T21:07:47.476 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.474+0000 7f158a598680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T21:07:47.480 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.478+0000 7f158a598680 5 note: GC not initialized 2023-10-05T21:07:47.482 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.479+0000 7f14bd7fa700 20 reqs_thread_entry: start 2023-10-05T21:07:47.482 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.479+0000 7f14b6ffd700 20 reqs_thread_entry: start 2023-10-05T21:07:47.482 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.479+0000 7f158a598680 20 init_complete bucket index max shards: 11 2023-10-05T21:07:47.482 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.479+0000 7f158a598680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T21:07:47.482 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f14b57fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 21:08:17 2023 2023-10-05T21:07:47.482 INFO:teuthology.orchestra.run.smithi070.stderr: 2023-10-05T21:07:47.482 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f14b4ff9700 20 reqs_thread_entry: start 2023-10-05T21:07:47.482 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 20 get_system_obj_state: rctx=0x7ffc67f011a8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x564a13654240 s->prefetch_data=0 2023-10-05T21:07:47.482 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T21:07:47.483 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2023-10-05T21:07:47.483 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T21:07:47.483 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 20 get_system_obj_state: rctx=0x7ffc67f00e98 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x564a13654240 s->prefetch_data=0 2023-10-05T21:07:47.483 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2023-10-05T21:07:47.484 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2023-10-05T21:07:47.484 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T21:07:47.484 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 20 get_system_obj_state: rctx=0x7ffc67f00e98 obj=default.rgw.meta:users.keys:XWTSCMQDUCFJOSOPVZAH state=0x564a13654240 s->prefetch_data=0 2023-10-05T21:07:47.484 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 10 cache get: name=default.rgw.meta+users.keys+XWTSCMQDUCFJOSOPVZAH : miss 2023-10-05T21:07:47.484 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 10 cache put: name=default.rgw.meta+users.keys+XWTSCMQDUCFJOSOPVZAH info.flags=0x0 2023-10-05T21:07:47.484 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 10 adding default.rgw.meta+users.keys+XWTSCMQDUCFJOSOPVZAH to cache LRU end 2023-10-05T21:07:47.484 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 20 get_system_obj_state: rctx=0x7ffc67f00728 obj=default.rgw.meta:users.keys:XWTSCMQDUCFJOSOPVZAH state=0x564a13654240 s->prefetch_data=0 2023-10-05T21:07:47.484 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 10 cache get: name=default.rgw.meta+users.keys+XWTSCMQDUCFJOSOPVZAH : hit (negative entry) 2023-10-05T21:07:47.485 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 20 get_system_obj_state: rctx=0x7ffc67f00dd8 obj=default.rgw.meta:users.keys:XWTSCMQDUCFJOSOPVZAH state=0x564a13654240 s->prefetch_data=0 2023-10-05T21:07:47.485 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.480+0000 7f158a598680 10 cache get: name=default.rgw.meta+users.keys+XWTSCMQDUCFJOSOPVZAH : hit (negative entry) 2023-10-05T21:07:47.485 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.481+0000 7f158a598680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T21:07:47.485 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.481+0000 7f158a598680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T21:07:47.485 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.481+0000 7f158a598680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T21:07:47.485 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.482+0000 7f156effd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084289 cookie 94876154585104 notifier 4222 bl.length()=640 2023-10-05T21:07:47.485 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.482+0000 7f156effd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T21:07:47.485 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.482+0000 7f156effd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T21:07:47.487 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.483+0000 7f158a598680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-10-05T21:07:47.488 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.483+0000 7f158a598680 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T21:07:47.488 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.483+0000 7f158a598680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-10-05T21:07:47.488 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.483+0000 7f156e7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084290 cookie 94876154585104 notifier 4222 bl.length()=194 2023-10-05T21:07:47.488 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.484+0000 7f156e7fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-10-05T21:07:47.488 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.484+0000 7f156e7fc700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T21:07:47.488 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.485+0000 7f158a598680 10 cache put: name=default.rgw.meta+users.keys+XWTSCMQDUCFJOSOPVZAH info.flags=0x7 2023-10-05T21:07:47.488 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.485+0000 7f158a598680 10 moving default.rgw.meta+users.keys+XWTSCMQDUCFJOSOPVZAH to cache LRU end 2023-10-05T21:07:47.488 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.485+0000 7f158a598680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:XWTSCMQDUCFJOSOPVZAH, ofs0, ns] 2023-10-05T21:07:47.488 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.485+0000 7f156effd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084291 cookie 94876154585104 notifier 4222 bl.length()=186 2023-10-05T21:07:47.488 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.485+0000 7f156effd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+XWTSCMQDUCFJOSOPVZAH info.flags=0x7 2023-10-05T21:07:47.488 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.485+0000 7f156effd700 10 rgw watcher librados: moving default.rgw.meta+users.keys+XWTSCMQDUCFJOSOPVZAH to cache LRU end 2023-10-05T21:07:47.489 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2023-10-05T21:07:47.489 INFO:teuthology.orchestra.run.smithi070.stdout: "user_id": "bar.client.0", 2023-10-05T21:07:47.489 INFO:teuthology.orchestra.run.smithi070.stdout: "display_name": "Mr. bar.client.0", 2023-10-05T21:07:47.489 INFO:teuthology.orchestra.run.smithi070.stdout: "email": "bar.client.0+test@test.test", 2023-10-05T21:07:47.489 INFO:teuthology.orchestra.run.smithi070.stdout: "suspended": 0, 2023-10-05T21:07:47.489 INFO:teuthology.orchestra.run.smithi070.stdout: "max_buckets": 1000, 2023-10-05T21:07:47.489 INFO:teuthology.orchestra.run.smithi070.stdout: "subusers": [], 2023-10-05T21:07:47.489 INFO:teuthology.orchestra.run.smithi070.stdout: "keys": [ 2023-10-05T21:07:47.489 INFO:teuthology.orchestra.run.smithi070.stdout: { 2023-10-05T21:07:47.490 INFO:teuthology.orchestra.run.smithi070.stdout: "user": "bar.client.0", 2023-10-05T21:07:47.490 INFO:teuthology.orchestra.run.smithi070.stdout: "access_key": "XWTSCMQDUCFJOSOPVZAH", 2023-10-05T21:07:47.490 INFO:teuthology.orchestra.run.smithi070.stdout: "secret_key": "QmI9XsLcWS13hBTmvXb/x6pFP2ZtCO+f7ZDICMpcfAyAqijKA3/lTA==" 2023-10-05T21:07:47.490 INFO:teuthology.orchestra.run.smithi070.stdout: } 2023-10-05T21:07:47.490 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2023-10-05T21:07:47.490 INFO:teuthology.orchestra.run.smithi070.stdout: "swift_keys": [], 2023-10-05T21:07:47.490 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": [ 2023-10-05T21:07:47.490 INFO:teuthology.orchestra.run.smithi070.stdout: { 2023-10-05T21:07:47.490 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "user-policy", 2023-10-05T21:07:47.490 INFO:teuthology.orchestra.run.smithi070.stdout: "perm": "*" 2023-10-05T21:07:47.490 INFO:teuthology.orchestra.run.smithi070.stdout: } 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: "op_mask": "read, write, delete", 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: "default_placement": "", 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: "default_storage_class": "", 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: "placement_tags": [], 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: "bucket_quota": { 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": false, 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: "check_on_raw": false, 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size": -1, 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size_kb": 0, 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: "max_objects": -1 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2023-10-05T21:07:47.491 INFO:teuthology.orchestra.run.smithi070.stdout: "user_quota": { 2023-10-05T21:07:47.492 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": false, 2023-10-05T21:07:47.492 INFO:teuthology.orchestra.run.smithi070.stdout: "check_on_raw": false, 2023-10-05T21:07:47.492 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size": -1, 2023-10-05T21:07:47.492 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size_kb": 0, 2023-10-05T21:07:47.492 INFO:teuthology.orchestra.run.smithi070.stdout: "max_objects": -1 2023-10-05T21:07:47.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2023-10-05T21:07:47.492 INFO:teuthology.orchestra.run.smithi070.stdout: "temp_url_keys": [], 2023-10-05T21:07:47.492 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "rgw", 2023-10-05T21:07:47.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mfa_ids": [] 2023-10-05T21:07:47.492 INFO:teuthology.orchestra.run.smithi070.stdout:} 2023-10-05T21:07:47.492 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:07:47.493 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.486+0000 7f158a598680 20 remove_watcher() i=0 2023-10-05T21:07:47.493 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.486+0000 7f158a598680 2 removed watcher, disabling cache 2023-10-05T21:07:47.493 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.487+0000 7f158a598680 20 remove_watcher() i=1 2023-10-05T21:07:47.493 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.488+0000 7f158a598680 20 remove_watcher() i=2 2023-10-05T21:07:47.493 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.489+0000 7f158a598680 20 remove_watcher() i=3 2023-10-05T21:07:47.494 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.489+0000 7f158a598680 20 remove_watcher() i=4 2023-10-05T21:07:47.494 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.490+0000 7f158a598680 20 remove_watcher() i=5 2023-10-05T21:07:47.494 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.491+0000 7f158a598680 20 remove_watcher() i=6 2023-10-05T21:07:47.494 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.492+0000 7f158a598680 20 remove_watcher() i=7 2023-10-05T21:07:47.494 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.492+0000 7f154e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:07:47.499 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 4964702371 --totp-seed JPXIP23SO4TDM4DEA46VYKSL4A7NTJHMUUTUL52K42PO6AQ3W3RGQ3ZE7BL4IUA7 --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T21:07:47.550 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setuser ceph since I am not root 2023-10-05T21:07:47.550 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T21:07:47.556 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.554+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.558 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.555+0000 7fecd2a04680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.558 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.555+0000 7fecd2a04680 20 realm 2023-10-05T21:07:47.558 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.555+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.558 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.555+0000 7fecd2a04680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.558 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.555+0000 7fecd2a04680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T21:07:47.558 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.555+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.558 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.556+0000 7fecd2a04680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.559 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.556+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.559 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.556+0000 7fecd2a04680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.559 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.556+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.559 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.556+0000 7fecd2a04680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:07:47.559 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.556+0000 7fecd2a04680 20 searching for the correct realm 2023-10-05T21:07:47.566 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.563+0000 7fecd2a04680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:07:47.566 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.563+0000 7fecd2a04680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:07:47.566 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.563+0000 7fecd2a04680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:07:47.566 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.563+0000 7fecd2a04680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:07:47.566 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.563+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.566 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.564+0000 7fecd2a04680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.567 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.564+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.567 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.564+0000 7fecd2a04680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.567 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.564+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.567 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.564+0000 7fecd2a04680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:07:47.567 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.564+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.567 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.564+0000 7fecd2a04680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.567 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.564+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.568 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.564+0000 7fecd2a04680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.568 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.564+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.568 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.565+0000 7fecd2a04680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.572 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.570+0000 7fecd2a04680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:07:47.573 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.570+0000 7fecd2a04680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:07:47.573 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.570+0000 7fecd2a04680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:07:47.573 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.570+0000 7fecd2a04680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:07:47.573 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.570+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.573 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.571+0000 7fecd2a04680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.573 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.571+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.575 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.571+0000 7fecd2a04680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:07:47.575 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.571+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.575 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.571+0000 7fecd2a04680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.575 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.571+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.575 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:07:47.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 zone default found 2023-10-05T21:07:47.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 4 Realm: () 2023-10-05T21:07:47.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 4 ZoneGroup: default (cb9d3d40-68a5-4791-bf47-2aada3a09714) 2023-10-05T21:07:47.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 4 Zone: default (bb6df8cb-740d-4b21-924c-79673a5d256e) 2023-10-05T21:07:47.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T21:07:47.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 zonegroup default 2023-10-05T21:07:47.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.577 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.577 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.577 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.577 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 started sync module instance, tier type = 2023-10-05T21:07:47.577 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.572+0000 7fecd2a04680 20 started zone id=bb6df8cb-740d-4b21-924c-79673a5d256e (name=default) with tier type = 2023-10-05T21:07:47.581 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.579+0000 7fecd2a04680 20 add_watcher() i=0 2023-10-05T21:07:47.582 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.580+0000 7fecd2a04680 20 add_watcher() i=1 2023-10-05T21:07:47.582 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.580+0000 7fecd2a04680 20 add_watcher() i=2 2023-10-05T21:07:47.582 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.580+0000 7fecd2a04680 20 add_watcher() i=3 2023-10-05T21:07:47.582 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.580+0000 7fecd2a04680 20 add_watcher() i=4 2023-10-05T21:07:47.582 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.580+0000 7fecd2a04680 20 add_watcher() i=5 2023-10-05T21:07:47.582 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.580+0000 7fecd2a04680 20 add_watcher() i=6 2023-10-05T21:07:47.582 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.580+0000 7fecd2a04680 20 add_watcher() i=7 2023-10-05T21:07:47.582 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.580+0000 7fecd2a04680 2 all 8 watchers are set, enabling cache 2023-10-05T21:07:47.583 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.581+0000 7fec977de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:07:47.583 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.581+0000 7fecd2a04680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T21:07:47.583 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.581+0000 7fecd2a04680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T21:07:47.587 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.585+0000 7fecd2a04680 5 note: GC not initialized 2023-10-05T21:07:47.589 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.586+0000 7fec067fc700 20 reqs_thread_entry: start 2023-10-05T21:07:47.589 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.586+0000 7febf7fff700 20 reqs_thread_entry: start 2023-10-05T21:07:47.589 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7fecd2a04680 20 init_complete bucket index max shards: 11 2023-10-05T21:07:47.589 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7fecd2a04680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T21:07:47.589 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7febf67fc700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 21:08:17 2023 2023-10-05T21:07:47.589 INFO:teuthology.orchestra.run.smithi070.stderr: 2023-10-05T21:07:47.589 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7febf5ffb700 20 reqs_thread_entry: start 2023-10-05T21:07:47.589 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7fecd2a04680 20 get_system_obj_state: rctx=0x7ffd04d462d8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55817e140240 s->prefetch_data=0 2023-10-05T21:07:47.589 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7fecd2a04680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T21:07:47.590 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7fecd2a04680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-10-05T21:07:47.590 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7fecd2a04680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T21:07:47.590 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7fecd2a04680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:07:47.590 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7fecd2a04680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T21:07:47.591 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7fecd2a04680 20 Read xattr: user.rgw.idtag 2023-10-05T21:07:47.591 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7fecd2a04680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T21:07:47.591 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.587+0000 7fecd2a04680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.591 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.588+0000 7fecd2a04680 20 rados_obj.operate() r=0 bl.length=455 2023-10-05T21:07:47.591 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.588+0000 7fecd2a04680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-10-05T21:07:47.591 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.588+0000 7fecd2a04680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T21:07:47.591 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.588+0000 7fecd2a04680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:07:47.591 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.589+0000 7fecd2a04680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T21:07:47.591 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.589+0000 7fecd2a04680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T21:07:47.592 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.589+0000 7fecd2a04680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T21:07:47.592 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.590+0000 7fecbf7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084292 cookie 94014655997440 notifier 4228 bl.length()=654 2023-10-05T21:07:47.592 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.590+0000 7fecbf7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T21:07:47.592 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.590+0000 7fecbf7fe700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T21:07:47.593 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.591+0000 7fecd2a04680 20 remove_watcher() i=0 2023-10-05T21:07:47.593 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.591+0000 7fecd2a04680 2 removed watcher, disabling cache 2023-10-05T21:07:47.594 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.592+0000 7fecd2a04680 20 remove_watcher() i=1 2023-10-05T21:07:47.594 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.593+0000 7fecd2a04680 20 remove_watcher() i=2 2023-10-05T21:07:47.595 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.594+0000 7fecd2a04680 20 remove_watcher() i=3 2023-10-05T21:07:47.596 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.594+0000 7fecd2a04680 20 remove_watcher() i=4 2023-10-05T21:07:47.597 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.595+0000 7fecd2a04680 20 remove_watcher() i=5 2023-10-05T21:07:47.597 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.596+0000 7fecd2a04680 20 remove_watcher() i=6 2023-10-05T21:07:47.598 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.597+0000 7fecd2a04680 20 remove_watcher() i=7 2023-10-05T21:07:47.599 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.597+0000 7fec977de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:07:47.604 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2023-10-05T21:07:47.604 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key MSXBHLLEJJIOCMIAIOBV --secret 1PhWWLQ4QVwa3Nkwwn6amIxVJs2HQBnVNHJpslCP9DVVwFXEPmvfRA== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2023-10-05T21:07:47.656 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setuser ceph since I am not root 2023-10-05T21:07:47.656 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T21:07:47.661 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.659+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.663 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.660+0000 7f28a7353680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.663 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.660+0000 7f28a7353680 20 realm 2023-10-05T21:07:47.663 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.660+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.663 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.661+0000 7f28a7353680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.663 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.661+0000 7f28a7353680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T21:07:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.661+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.661+0000 7f28a7353680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.661+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.661+0000 7f28a7353680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.661+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.661+0000 7f28a7353680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:07:47.664 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.661+0000 7f28a7353680 20 searching for the correct realm 2023-10-05T21:07:47.671 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.668+0000 7f28a7353680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:07:47.671 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.668+0000 7f28a7353680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:07:47.671 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.668+0000 7f28a7353680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:07:47.671 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.668+0000 7f28a7353680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:07:47.671 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.668+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.671 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.669+0000 7f28a7353680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.671 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.669+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.671 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.669+0000 7f28a7353680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.671 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.669+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.672 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.669+0000 7f28a7353680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:07:47.672 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.669+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.672 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.669+0000 7f28a7353680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.672 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.669+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.672 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.669+0000 7f28a7353680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.673 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.669+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.673 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.670+0000 7f28a7353680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.677 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.675+0000 7f28a7353680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:07:47.677 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.675+0000 7f28a7353680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:07:47.677 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.675+0000 7f28a7353680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:07:47.678 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.675+0000 7f28a7353680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:07:47.678 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.675+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.678 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.678 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.679 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:07:47.679 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.680 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.680 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.680 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:07:47.680 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.680 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.680 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 20 zone default found 2023-10-05T21:07:47.680 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 4 Realm: () 2023-10-05T21:07:47.680 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 4 ZoneGroup: default (cb9d3d40-68a5-4791-bf47-2aada3a09714) 2023-10-05T21:07:47.680 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 4 Zone: default (bb6df8cb-740d-4b21-924c-79673a5d256e) 2023-10-05T21:07:47.680 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T21:07:47.680 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 20 zonegroup default 2023-10-05T21:07:47.680 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.676+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.681 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.677+0000 7f28a7353680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.681 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.677+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.681 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.677+0000 7f28a7353680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.681 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.677+0000 7f28a7353680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.681 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.677+0000 7f28a7353680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.681 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.677+0000 7f28a7353680 20 started sync module instance, tier type = 2023-10-05T21:07:47.681 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.677+0000 7f28a7353680 20 started zone id=bb6df8cb-740d-4b21-924c-79673a5d256e (name=default) with tier type = 2023-10-05T21:07:47.686 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.684+0000 7f28a7353680 20 add_watcher() i=0 2023-10-05T21:07:47.686 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.684+0000 7f28a7353680 20 add_watcher() i=1 2023-10-05T21:07:47.686 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.684+0000 7f28a7353680 20 add_watcher() i=2 2023-10-05T21:07:47.686 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.684+0000 7f28a7353680 20 add_watcher() i=3 2023-10-05T21:07:47.686 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.684+0000 7f28a7353680 20 add_watcher() i=4 2023-10-05T21:07:47.686 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.684+0000 7f28a7353680 20 add_watcher() i=5 2023-10-05T21:07:47.686 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.684+0000 7f28a7353680 20 add_watcher() i=6 2023-10-05T21:07:47.687 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.685+0000 7f28a7353680 20 add_watcher() i=7 2023-10-05T21:07:47.687 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.685+0000 7f28a7353680 2 all 8 watchers are set, enabling cache 2023-10-05T21:07:47.688 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.686+0000 7f286bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:07:47.688 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.686+0000 7f28a7353680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T21:07:47.688 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.686+0000 7f28a7353680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T21:07:47.692 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.690+0000 7f28a7353680 5 note: GC not initialized 2023-10-05T21:07:47.694 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.691+0000 7f27daffd700 20 reqs_thread_entry: start 2023-10-05T21:07:47.694 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.691+0000 7f27d8ff9700 20 reqs_thread_entry: start 2023-10-05T21:07:47.694 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.691+0000 7f28a7353680 20 init_complete bucket index max shards: 11 2023-10-05T21:07:47.694 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.691+0000 7f28a7353680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T21:07:47.694 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.691+0000 7f27ceffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 21:08:18 2023 2023-10-05T21:07:47.694 INFO:teuthology.orchestra.run.smithi070.stderr: 2023-10-05T21:07:47.695 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.691+0000 7f27ce7fc700 20 reqs_thread_entry: start 2023-10-05T21:07:47.695 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.691+0000 7f28a7353680 20 get_system_obj_state: rctx=0x7ffca2a38c88 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x5579f1d73240 s->prefetch_data=0 2023-10-05T21:07:47.695 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.691+0000 7f28a7353680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T21:07:47.695 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2023-10-05T21:07:47.695 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T21:07:47.695 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 20 get_system_obj_state: rctx=0x7ffca2a38978 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x5579f1d73240 s->prefetch_data=0 2023-10-05T21:07:47.695 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2023-10-05T21:07:47.695 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2023-10-05T21:07:47.695 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T21:07:47.695 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 20 get_system_obj_state: rctx=0x7ffca2a38978 obj=default.rgw.meta:users.keys:MSXBHLLEJJIOCMIAIOBV state=0x5579f1d73240 s->prefetch_data=0 2023-10-05T21:07:47.695 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 10 cache get: name=default.rgw.meta+users.keys+MSXBHLLEJJIOCMIAIOBV : miss 2023-10-05T21:07:47.696 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 10 cache put: name=default.rgw.meta+users.keys+MSXBHLLEJJIOCMIAIOBV info.flags=0x0 2023-10-05T21:07:47.696 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 10 adding default.rgw.meta+users.keys+MSXBHLLEJJIOCMIAIOBV to cache LRU end 2023-10-05T21:07:47.696 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 20 get_system_obj_state: rctx=0x7ffca2a38208 obj=default.rgw.meta:users.keys:MSXBHLLEJJIOCMIAIOBV state=0x5579f1d73240 s->prefetch_data=0 2023-10-05T21:07:47.696 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 10 cache get: name=default.rgw.meta+users.keys+MSXBHLLEJJIOCMIAIOBV : hit (negative entry) 2023-10-05T21:07:47.696 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 20 get_system_obj_state: rctx=0x7ffca2a388b8 obj=default.rgw.meta:users.keys:MSXBHLLEJJIOCMIAIOBV state=0x5579f1d73240 s->prefetch_data=0 2023-10-05T21:07:47.696 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.692+0000 7f28a7353680 10 cache get: name=default.rgw.meta+users.keys+MSXBHLLEJJIOCMIAIOBV : hit (negative entry) 2023-10-05T21:07:47.697 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.693+0000 7f28a7353680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T21:07:47.697 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.693+0000 7f28a7353680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T21:07:47.697 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.693+0000 7f28a7353680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T21:07:47.697 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.693+0000 7f288bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084291 cookie 93982238444624 notifier 4234 bl.length()=714 2023-10-05T21:07:47.697 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.693+0000 7f288bfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T21:07:47.697 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.693+0000 7f288bfff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T21:07:47.698 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.695+0000 7f28a7353680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-10-05T21:07:47.698 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.695+0000 7f28a7353680 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T21:07:47.698 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.695+0000 7f28a7353680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-10-05T21:07:47.698 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.695+0000 7f288b7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084292 cookie 93982238448304 notifier 4234 bl.length()=224 2023-10-05T21:07:47.698 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.695+0000 7f288b7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-10-05T21:07:47.698 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.695+0000 7f288b7fe700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T21:07:47.699 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.696+0000 7f28a7353680 10 cache put: name=default.rgw.meta+users.keys+MSXBHLLEJJIOCMIAIOBV info.flags=0x7 2023-10-05T21:07:47.699 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.696+0000 7f28a7353680 10 moving default.rgw.meta+users.keys+MSXBHLLEJJIOCMIAIOBV to cache LRU end 2023-10-05T21:07:47.699 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.696+0000 7f28a7353680 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.keys:MSXBHLLEJJIOCMIAIOBV, ofs0, ns] 2023-10-05T21:07:47.699 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.696+0000 7f288bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084293 cookie 93982238434832 notifier 4234 bl.length()=201 2023-10-05T21:07:47.699 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.696+0000 7f288bfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+MSXBHLLEJJIOCMIAIOBV info.flags=0x7 2023-10-05T21:07:47.699 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.696+0000 7f288bfff700 10 rgw watcher librados: moving default.rgw.meta+users.keys+MSXBHLLEJJIOCMIAIOBV to cache LRU end 2023-10-05T21:07:47.699 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2023-10-05T21:07:47.700 INFO:teuthology.orchestra.run.smithi070.stdout: "user_id": "testx$tenanteduser.client.0", 2023-10-05T21:07:47.700 INFO:teuthology.orchestra.run.smithi070.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2023-10-05T21:07:47.700 INFO:teuthology.orchestra.run.smithi070.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2023-10-05T21:07:47.700 INFO:teuthology.orchestra.run.smithi070.stdout: "suspended": 0, 2023-10-05T21:07:47.700 INFO:teuthology.orchestra.run.smithi070.stdout: "max_buckets": 1000, 2023-10-05T21:07:47.700 INFO:teuthology.orchestra.run.smithi070.stdout: "subusers": [], 2023-10-05T21:07:47.700 INFO:teuthology.orchestra.run.smithi070.stdout: "keys": [ 2023-10-05T21:07:47.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2023-10-05T21:07:47.700 INFO:teuthology.orchestra.run.smithi070.stdout: "user": "testx$tenanteduser.client.0", 2023-10-05T21:07:47.700 INFO:teuthology.orchestra.run.smithi070.stdout: "access_key": "MSXBHLLEJJIOCMIAIOBV", 2023-10-05T21:07:47.701 INFO:teuthology.orchestra.run.smithi070.stdout: "secret_key": "1PhWWLQ4QVwa3Nkwwn6amIxVJs2HQBnVNHJpslCP9DVVwFXEPmvfRA==" 2023-10-05T21:07:47.701 INFO:teuthology.orchestra.run.smithi070.stdout: } 2023-10-05T21:07:47.701 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2023-10-05T21:07:47.701 INFO:teuthology.orchestra.run.smithi070.stdout: "swift_keys": [], 2023-10-05T21:07:47.701 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": [ 2023-10-05T21:07:47.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2023-10-05T21:07:47.701 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "user-policy", 2023-10-05T21:07:47.701 INFO:teuthology.orchestra.run.smithi070.stdout: "perm": "*" 2023-10-05T21:07:47.701 INFO:teuthology.orchestra.run.smithi070.stdout: } 2023-10-05T21:07:47.701 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2023-10-05T21:07:47.701 INFO:teuthology.orchestra.run.smithi070.stdout: "op_mask": "read, write, delete", 2023-10-05T21:07:47.702 INFO:teuthology.orchestra.run.smithi070.stdout: "default_placement": "", 2023-10-05T21:07:47.702 INFO:teuthology.orchestra.run.smithi070.stdout: "default_storage_class": "", 2023-10-05T21:07:47.702 INFO:teuthology.orchestra.run.smithi070.stdout: "placement_tags": [], 2023-10-05T21:07:47.702 INFO:teuthology.orchestra.run.smithi070.stdout: "bucket_quota": { 2023-10-05T21:07:47.702 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": false, 2023-10-05T21:07:47.702 INFO:teuthology.orchestra.run.smithi070.stdout: "check_on_raw": false, 2023-10-05T21:07:47.702 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size": -1, 2023-10-05T21:07:47.702 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size_kb": 0, 2023-10-05T21:07:47.702 INFO:teuthology.orchestra.run.smithi070.stdout: "max_objects": -1 2023-10-05T21:07:47.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2023-10-05T21:07:47.702 INFO:teuthology.orchestra.run.smithi070.stdout: "user_quota": { 2023-10-05T21:07:47.703 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": false, 2023-10-05T21:07:47.703 INFO:teuthology.orchestra.run.smithi070.stdout: "check_on_raw": false, 2023-10-05T21:07:47.703 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size": -1, 2023-10-05T21:07:47.703 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size_kb": 0, 2023-10-05T21:07:47.703 INFO:teuthology.orchestra.run.smithi070.stdout: "max_objects": -1 2023-10-05T21:07:47.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2023-10-05T21:07:47.703 INFO:teuthology.orchestra.run.smithi070.stdout: "temp_url_keys": [], 2023-10-05T21:07:47.703 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "rgw", 2023-10-05T21:07:47.703 INFO:teuthology.orchestra.run.smithi070.stdout: "mfa_ids": [] 2023-10-05T21:07:47.703 INFO:teuthology.orchestra.run.smithi070.stdout:} 2023-10-05T21:07:47.703 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:07:47.704 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.698+0000 7f28a7353680 20 remove_watcher() i=0 2023-10-05T21:07:47.704 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.698+0000 7f28a7353680 2 removed watcher, disabling cache 2023-10-05T21:07:47.704 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.699+0000 7f28a7353680 20 remove_watcher() i=1 2023-10-05T21:07:47.704 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.699+0000 7f28a7353680 20 remove_watcher() i=2 2023-10-05T21:07:47.704 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.700+0000 7f28a7353680 20 remove_watcher() i=3 2023-10-05T21:07:47.705 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.701+0000 7f28a7353680 20 remove_watcher() i=4 2023-10-05T21:07:47.705 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.702+0000 7f28a7353680 20 remove_watcher() i=5 2023-10-05T21:07:47.705 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.703+0000 7f28a7353680 20 remove_watcher() i=6 2023-10-05T21:07:47.705 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.703+0000 7f28a7353680 20 remove_watcher() i=7 2023-10-05T21:07:47.706 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.704+0000 7f286bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:07:47.710 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 4360966520 --totp-seed J5Y34GY2P5BTGQVRE3TW34GKXQJNPAYUEKHMWOMNSSOAZGMYFBDACQPXKECEOBJZ --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T21:07:47.761 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setuser ceph since I am not root 2023-10-05T21:07:47.762 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T21:07:47.767 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.765+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.769 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.766+0000 7fcd71734680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.769 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.766+0000 7fcd71734680 20 realm 2023-10-05T21:07:47.769 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.766+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.769 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.766+0000 7fcd71734680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.769 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.766+0000 7fcd71734680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T21:07:47.769 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.766+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.769 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.767+0000 7fcd71734680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.769 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.767+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.769 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.767+0000 7fcd71734680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.769 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.767+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.770 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.767+0000 7fcd71734680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:07:47.770 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.767+0000 7fcd71734680 20 searching for the correct realm 2023-10-05T21:07:47.777 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.774+0000 7fcd71734680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:07:47.777 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.774+0000 7fcd71734680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:07:47.777 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.774+0000 7fcd71734680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:07:47.777 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.774+0000 7fcd71734680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:07:47.777 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.774+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.777 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.774+0000 7fcd71734680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.777 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.774+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.778 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.775+0000 7fcd71734680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.778 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.775+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.778 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.775+0000 7fcd71734680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:07:47.778 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.775+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.778 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.775+0000 7fcd71734680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.778 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.775+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.778 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.775+0000 7fcd71734680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.778 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.775+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.779 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.775+0000 7fcd71734680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.784 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.781+0000 7fcd71734680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:07:47.784 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.781+0000 7fcd71734680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:07:47.784 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.781+0000 7fcd71734680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:07:47.784 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.781+0000 7fcd71734680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:07:47.784 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.781+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.781+0000 7fcd71734680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.781+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:07:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:07:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:07:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 20 zone default found 2023-10-05T21:07:47.785 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 4 Realm: () 2023-10-05T21:07:47.786 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 4 ZoneGroup: default (cb9d3d40-68a5-4791-bf47-2aada3a09714) 2023-10-05T21:07:47.786 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 4 Zone: default (bb6df8cb-740d-4b21-924c-79673a5d256e) 2023-10-05T21:07:47.786 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T21:07:47.786 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 20 zonegroup default 2023-10-05T21:07:47.786 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.786 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.786 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.782+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.787 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.783+0000 7fcd71734680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.787 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.783+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.787 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.783+0000 7fcd71734680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:07:47.787 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.783+0000 7fcd71734680 20 started sync module instance, tier type = 2023-10-05T21:07:47.787 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.783+0000 7fcd71734680 20 started zone id=bb6df8cb-740d-4b21-924c-79673a5d256e (name=default) with tier type = 2023-10-05T21:07:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.790+0000 7fcd71734680 20 add_watcher() i=0 2023-10-05T21:07:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.790+0000 7fcd71734680 20 add_watcher() i=1 2023-10-05T21:07:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.790+0000 7fcd71734680 20 add_watcher() i=2 2023-10-05T21:07:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.790+0000 7fcd71734680 20 add_watcher() i=3 2023-10-05T21:07:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.790+0000 7fcd71734680 20 add_watcher() i=4 2023-10-05T21:07:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.790+0000 7fcd71734680 20 add_watcher() i=5 2023-10-05T21:07:47.792 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.790+0000 7fcd71734680 20 add_watcher() i=6 2023-10-05T21:07:47.793 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.791+0000 7fcd71734680 20 add_watcher() i=7 2023-10-05T21:07:47.793 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.791+0000 7fcd71734680 2 all 8 watchers are set, enabling cache 2023-10-05T21:07:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.792+0000 7fcd357ea700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:07:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.792+0000 7fcd71734680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T21:07:47.794 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.792+0000 7fcd71734680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T21:07:47.797 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.796+0000 7fcd71734680 5 note: GC not initialized 2023-10-05T21:07:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.796+0000 7fcca3fff700 20 reqs_thread_entry: start 2023-10-05T21:07:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcca1ffb700 20 reqs_thread_entry: start 2023-10-05T21:07:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcd71734680 20 init_complete bucket index max shards: 11 2023-10-05T21:07:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcd71734680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T21:07:47.799 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcc8ffff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 21:08:18 2023 2023-10-05T21:07:47.799 INFO:teuthology.orchestra.run.smithi070.stderr: 2023-10-05T21:07:47.800 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcc8f7fe700 20 reqs_thread_entry: start 2023-10-05T21:07:47.800 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcd71734680 20 get_system_obj_state: rctx=0x7ffe215fc798 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x561eda7e9240 s->prefetch_data=0 2023-10-05T21:07:47.800 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcd71734680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T21:07:47.800 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcd71734680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-10-05T21:07:47.800 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcd71734680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T21:07:47.801 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcd71734680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:07:47.801 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcd71734680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T21:07:47.801 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcd71734680 20 Read xattr: user.rgw.idtag 2023-10-05T21:07:47.801 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcd71734680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T21:07:47.801 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.797+0000 7fcd71734680 20 rados->read ofs=0 len=0 2023-10-05T21:07:47.801 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.798+0000 7fcd71734680 20 rados_obj.operate() r=0 bl.length=514 2023-10-05T21:07:47.801 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.798+0000 7fcd71734680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-10-05T21:07:47.801 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.798+0000 7fcd71734680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T21:07:47.801 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.798+0000 7fcd71734680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:07:47.802 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.800+0000 7fcd71734680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T21:07:47.802 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.800+0000 7fcd71734680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T21:07:47.802 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.800+0000 7fcd71734680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T21:07:47.802 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.800+0000 7fcd5e7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084293 cookie 94690516358416 notifier 4240 bl.length()=728 2023-10-05T21:07:47.802 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.800+0000 7fcd5e7fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T21:07:47.803 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.800+0000 7fcd5e7fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T21:07:47.803 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.802+0000 7fcd71734680 20 remove_watcher() i=0 2023-10-05T21:07:47.803 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.802+0000 7fcd71734680 2 removed watcher, disabling cache 2023-10-05T21:07:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.802+0000 7fcd71734680 20 remove_watcher() i=1 2023-10-05T21:07:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.803+0000 7fcd71734680 20 remove_watcher() i=2 2023-10-05T21:07:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.804+0000 7fcd71734680 20 remove_watcher() i=3 2023-10-05T21:07:47.806 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.805+0000 7fcd71734680 20 remove_watcher() i=4 2023-10-05T21:07:47.807 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.806+0000 7fcd71734680 20 remove_watcher() i=5 2023-10-05T21:07:47.808 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.806+0000 7fcd71734680 20 remove_watcher() i=6 2023-10-05T21:07:47.809 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.807+0000 7fcd71734680 20 remove_watcher() i=7 2023-10-05T21:07:47.810 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:07:47.808+0000 7fcd357ea700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:07:47.814 INFO:tasks.s3tests:Configuring s3-tests... 2023-10-05T21:07:47.814 DEBUG:teuthology.orchestra.run.smithi070:> cd /home/ubuntu/cephtest/s3-tests-client.0 && ./bootstrap 2023-10-05T21:07:56.870 INFO:teuthology.orchestra.run.smithi070.stderr:./bootstrap: Missing required RPM packages: libevent-devel libffi-devel libxml2-devel libxslt-devel zlib-devel. 2023-10-05T21:07:57.713 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:40:19 ago on Thu 05 Oct 2023 08:27:38 PM UTC. 2023-10-05T21:07:59.298 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-10-05T21:07:59.299 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:07:59.300 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-10-05T21:07:59.300 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:07:59.300 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-10-05T21:07:59.300 INFO:teuthology.orchestra.run.smithi070.stdout: libevent-devel x86_64 2.1.8-5.el8 appstream 104 k 2023-10-05T21:07:59.300 INFO:teuthology.orchestra.run.smithi070.stdout: libffi-devel x86_64 3.1-24.el8 baseos 29 k 2023-10-05T21:07:59.300 INFO:teuthology.orchestra.run.smithi070.stdout: libxml2-devel x86_64 2.9.7-16.el8 appstream 1.0 M 2023-10-05T21:07:59.300 INFO:teuthology.orchestra.run.smithi070.stdout: libxslt-devel x86_64 1.1.32-6.el8 appstream 323 k 2023-10-05T21:07:59.300 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel x86_64 1.2.11-25.el8 baseos 59 k 2023-10-05T21:07:59.300 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2023-10-05T21:07:59.300 INFO:teuthology.orchestra.run.smithi070.stdout: zlib x86_64 1.2.11-25.el8 baseos 103 k 2023-10-05T21:07:59.300 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2023-10-05T21:07:59.300 INFO:teuthology.orchestra.run.smithi070.stdout: cmake-filesystem x86_64 3.20.2-5.el8 appstream 45 k 2023-10-05T21:07:59.301 INFO:teuthology.orchestra.run.smithi070.stdout: libgcrypt-devel x86_64 1.8.5-7.el8 baseos 150 k 2023-10-05T21:07:59.301 INFO:teuthology.orchestra.run.smithi070.stdout: libgpg-error-devel x86_64 1.31-1.el8 baseos 64 k 2023-10-05T21:07:59.301 INFO:teuthology.orchestra.run.smithi070.stdout: xz-devel x86_64 5.2.4-4.el8 baseos 63 k 2023-10-05T21:07:59.301 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:07:59.301 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-10-05T21:07:59.301 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-10-05T21:07:59.301 INFO:teuthology.orchestra.run.smithi070.stdout:Install 9 Packages 2023-10-05T21:07:59.301 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 1 Package 2023-10-05T21:07:59.301 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:07:59.302 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 2.0 M 2023-10-05T21:07:59.302 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-10-05T21:07:59.584 INFO:teuthology.orchestra.run.smithi070.stdout:(1/10): cmake-filesystem-3.20.2-5.el8.x86_64.rp 221 kB/s | 45 kB 00:00 2023-10-05T21:07:59.792 INFO:teuthology.orchestra.run.smithi070.stdout:(2/10): libevent-devel-2.1.8-5.el8.x86_64.rpm 254 kB/s | 104 kB 00:00 2023-10-05T21:07:59.843 INFO:teuthology.orchestra.run.smithi070.stdout:(3/10): libffi-devel-3.1-24.el8.x86_64.rpm 581 kB/s | 29 kB 00:00 2023-10-05T21:07:59.893 INFO:teuthology.orchestra.run.smithi070.stdout:(4/10): libgcrypt-devel-1.8.5-7.el8.x86_64.rpm 2.9 MB/s | 150 kB 00:00 2023-10-05T21:07:59.927 INFO:teuthology.orchestra.run.smithi070.stdout:(5/10): libgpg-error-devel-1.31-1.el8.x86_64.rp 1.9 MB/s | 64 kB 00:00 2023-10-05T21:07:59.960 INFO:teuthology.orchestra.run.smithi070.stdout:(6/10): xz-devel-5.2.4-4.el8.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-05T21:07:59.994 INFO:teuthology.orchestra.run.smithi070.stdout:(7/10): zlib-devel-1.2.11-25.el8.x86_64.rpm 1.7 MB/s | 59 kB 00:00 2023-10-05T21:08:00.027 INFO:teuthology.orchestra.run.smithi070.stdout:(8/10): zlib-1.2.11-25.el8.x86_64.rpm 3.0 MB/s | 103 kB 00:00 2023-10-05T21:08:00.078 INFO:teuthology.orchestra.run.smithi070.stdout:(9/10): libxslt-devel-1.1.32-6.el8.x86_64.rpm 654 kB/s | 323 kB 00:00 2023-10-05T21:08:00.945 INFO:teuthology.orchestra.run.smithi070.stdout:(10/10): libxml2-devel-2.9.7-16.el8.x86_64.rpm 682 kB/s | 1.0 MB 00:01 2023-10-05T21:08:00.945 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-10-05T21:08:00.945 INFO:teuthology.orchestra.run.smithi070.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2023-10-05T21:08:01.046 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-10-05T21:08:01.176 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-10-05T21:08:01.176 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-10-05T21:08:01.467 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-10-05T21:08:01.470 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-10-05T21:08:02.104 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-10-05T21:08:02.221 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 1/11 2023-10-05T21:08:02.254 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 1/11 2023-10-05T21:08:02.456 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgcrypt-devel-1.8.5-7.el8.x86_64 2/11 2023-10-05T21:08:02.473 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libgcrypt-devel-1.8.5-7.el8.x86_64 2/11 2023-10-05T21:08:02.668 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : zlib-1.2.11-25.el8.x86_64 3/11 2023-10-05T21:08:02.859 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : zlib-devel-1.2.11-25.el8.x86_64 4/11 2023-10-05T21:08:03.050 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xz-devel-5.2.4-4.el8.x86_64 5/11 2023-10-05T21:08:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cmake-filesystem-3.20.2-5.el8.x86_64 6/11 2023-10-05T21:08:03.827 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libxml2-devel-2.9.7-16.el8.x86_64 7/11 2023-10-05T21:08:04.042 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 8/11 2023-10-05T21:08:04.196 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libffi-devel-3.1-24.el8.x86_64 9/11 2023-10-05T21:08:04.224 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libffi-devel-3.1-24.el8.x86_64 9/11 2023-10-05T21:08:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 10/11 2023-10-05T21:08:04.595 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : zlib-1.2.11-21.el8.x86_64 11/11 2023-10-05T21:08:04.907 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: zlib-1.2.11-21.el8.x86_64 11/11 2023-10-05T21:08:04.907 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cmake-filesystem-3.20.2-5.el8.x86_64 1/11 2023-10-05T21:08:04.907 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 2/11 2023-10-05T21:08:04.907 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libxml2-devel-2.9.7-16.el8.x86_64 3/11 2023-10-05T21:08:04.907 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 4/11 2023-10-05T21:08:04.907 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libffi-devel-3.1-24.el8.x86_64 5/11 2023-10-05T21:08:04.907 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgcrypt-devel-1.8.5-7.el8.x86_64 6/11 2023-10-05T21:08:04.907 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 7/11 2023-10-05T21:08:04.908 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xz-devel-5.2.4-4.el8.x86_64 8/11 2023-10-05T21:08:04.908 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : zlib-devel-1.2.11-25.el8.x86_64 9/11 2023-10-05T21:08:04.908 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : zlib-1.2.11-25.el8.x86_64 10/11 2023-10-05T21:08:05.628 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : zlib-1.2.11-21.el8.x86_64 11/11 2023-10-05T21:08:05.628 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:05.628 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2023-10-05T21:08:05.628 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-1.2.11-25.el8.x86_64 2023-10-05T21:08:05.629 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-10-05T21:08:05.629 INFO:teuthology.orchestra.run.smithi070.stdout: cmake-filesystem-3.20.2-5.el8.x86_64 libevent-devel-2.1.8-5.el8.x86_64 2023-10-05T21:08:05.629 INFO:teuthology.orchestra.run.smithi070.stdout: libffi-devel-3.1-24.el8.x86_64 libgcrypt-devel-1.8.5-7.el8.x86_64 2023-10-05T21:08:05.629 INFO:teuthology.orchestra.run.smithi070.stdout: libgpg-error-devel-1.31-1.el8.x86_64 libxml2-devel-2.9.7-16.el8.x86_64 2023-10-05T21:08:05.629 INFO:teuthology.orchestra.run.smithi070.stdout: libxslt-devel-1.1.32-6.el8.x86_64 xz-devel-5.2.4-4.el8.x86_64 2023-10-05T21:08:05.629 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel-1.2.11-25.el8.x86_64 2023-10-05T21:08:05.629 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:05.629 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-10-05T21:08:06.189 INFO:teuthology.orchestra.run.smithi070.stdout:Already using interpreter /usr/bin/python3.6 2023-10-05T21:08:06.189 INFO:teuthology.orchestra.run.smithi070.stdout:Using base prefix '/usr' 2023-10-05T21:08:06.189 INFO:teuthology.orchestra.run.smithi070.stdout:New python executable in /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python3.6 2023-10-05T21:08:06.189 INFO:teuthology.orchestra.run.smithi070.stdout:Also creating executable in /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python 2023-10-05T21:08:09.536 INFO:teuthology.orchestra.run.smithi070.stdout:Installing setuptools, pip, wheel...done. 2023-10-05T21:08:09.879 INFO:teuthology.orchestra.run.smithi070.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.6/site-packages (21.3.1) 2023-10-05T21:08:11.110 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting setuptools==44.1.0 2023-10-05T21:08:11.168 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2023-10-05T21:08:11.298 INFO:teuthology.orchestra.run.smithi070.stdout:Installing collected packages: setuptools 2023-10-05T21:08:11.299 INFO:teuthology.orchestra.run.smithi070.stdout: Attempting uninstall: setuptools 2023-10-05T21:08:11.299 INFO:teuthology.orchestra.run.smithi070.stdout: Found existing installation: setuptools 59.6.0 2023-10-05T21:08:11.346 INFO:teuthology.orchestra.run.smithi070.stdout: Uninstalling setuptools-59.6.0: 2023-10-05T21:08:11.355 INFO:teuthology.orchestra.run.smithi070.stdout: Successfully uninstalled setuptools-59.6.0 2023-10-05T21:08:11.576 INFO:teuthology.orchestra.run.smithi070.stdout:Successfully installed setuptools-44.1.0 2023-10-05T21:08:12.165 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting PyYAML 2023-10-05T21:08:12.214 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB) 2023-10-05T21:08:12.340 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting nose>=1.0.0 2023-10-05T21:08:12.355 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2023-10-05T21:08:12.453 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting boto>=2.6.0 2023-10-05T21:08:12.472 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2023-10-05T21:08:13.456 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting boto3>=1.0.0 2023-10-05T21:08:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2023-10-05T21:08:14.568 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting botocore<1.28.0 2023-10-05T21:08:14.585 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2023-10-05T21:08:15.094 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting munch>=2.0.0 2023-10-05T21:08:15.110 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading munch-4.0.0-py2.py3-none-any.whl (9.9 kB) 2023-10-05T21:08:15.565 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting gevent>=1.0 2023-10-05T21:08:15.584 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading gevent-22.10.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB) 2023-10-05T21:08:15.770 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting isodate>=0.4.4 2023-10-05T21:08:15.784 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2023-10-05T21:08:15.911 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting requests>=2.23.0 2023-10-05T21:08:15.926 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2023-10-05T21:08:16.112 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting pytz>=2011k 2023-10-05T21:08:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) 2023-10-05T21:08:16.235 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting httplib2 2023-10-05T21:08:16.251 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading httplib2-0.22.0-py3-none-any.whl (96 kB) 2023-10-05T21:08:16.790 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting lxml 2023-10-05T21:08:16.811 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading lxml-4.9.3-cp36-cp36m-manylinux_2_28_x86_64.whl (7.4 MB) 2023-10-05T21:08:17.155 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2023-10-05T21:08:17.170 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2023-10-05T21:08:17.246 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting jmespath<2.0.0,>=0.7.1 2023-10-05T21:08:17.261 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2023-10-05T21:08:17.408 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting urllib3<1.27,>=1.25.4 2023-10-05T21:08:17.424 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading urllib3-1.26.17-py2.py3-none-any.whl (143 kB) 2023-10-05T21:08:17.522 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting python-dateutil<3.0.0,>=2.1 2023-10-05T21:08:17.538 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-10-05T21:08:17.738 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting importlib-metadata>=1.7.0 2023-10-05T21:08:17.753 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB) 2023-10-05T21:08:18.268 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting greenlet>=2.0.0 2023-10-05T21:08:18.293 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading greenlet-2.0.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (564 kB) 2023-10-05T21:08:18.398 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting zope.event 2023-10-05T21:08:18.413 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB) 2023-10-05T21:08:18.436 INFO:teuthology.orchestra.run.smithi070.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (from gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2023-10-05T21:08:18.812 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting zope.interface 2023-10-05T21:08:18.827 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253 kB) 2023-10-05T21:08:18.914 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting six 2023-10-05T21:08:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-10-05T21:08:19.189 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting charset-normalizer~=2.0.0 2023-10-05T21:08:19.205 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2023-10-05T21:08:19.343 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting certifi>=2017.4.17 2023-10-05T21:08:19.359 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading certifi-2023.7.22-py3-none-any.whl (158 kB) 2023-10-05T21:08:19.467 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting idna<4,>=2.5 2023-10-05T21:08:19.481 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading idna-3.4-py3-none-any.whl (61 kB) 2023-10-05T21:08:19.625 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2023-10-05T21:08:19.641 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB) 2023-10-05T21:08:19.788 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting zipp>=0.5 2023-10-05T21:08:19.803 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB) 2023-10-05T21:08:19.892 INFO:teuthology.orchestra.run.smithi070.stdout:Collecting typing-extensions>=3.6.4 2023-10-05T21:08:19.907 INFO:teuthology.orchestra.run.smithi070.stdout: Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB) 2023-10-05T21:08:20.280 INFO:teuthology.orchestra.run.smithi070.stdout:Installing collected packages: six, urllib3, python-dateutil, jmespath, zipp, typing-extensions, botocore, zope.interface, zope.event, s3transfer, pyparsing, importlib-metadata, idna, greenlet, charset-normalizer, certifi, requests, PyYAML, pytz, nose, munch, lxml, isodate, httplib2, gevent, boto3, boto 2023-10-05T21:08:23.677 INFO:teuthology.orchestra.run.smithi070.stdout:Successfully installed PyYAML-6.0.1 boto-2.49.0 boto3-1.23.10 botocore-1.26.10 certifi-2023.7.22 charset-normalizer-2.0.12 gevent-22.10.2 greenlet-2.0.2 httplib2-0.22.0 idna-3.4 importlib-metadata-4.8.3 isodate-0.6.1 jmespath-0.10.0 lxml-4.9.3 munch-4.0.0 nose-1.3.7 pyparsing-3.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 requests-2.27.1 s3transfer-0.5.2 six-1.16.0 typing-extensions-4.1.1 urllib3-1.26.17 zipp-3.6.0 zope.event-4.6 zope.interface-5.5.2 2023-10-05T21:08:23.984 INFO:teuthology.orchestra.run.smithi070.stdout:running develop 2023-10-05T21:08:24.005 INFO:teuthology.orchestra.run.smithi070.stdout:running egg_info 2023-10-05T21:08:24.005 INFO:teuthology.orchestra.run.smithi070.stdout:creating s3tests.egg-info 2023-10-05T21:08:24.005 INFO:teuthology.orchestra.run.smithi070.stdout:writing s3tests.egg-info/PKG-INFO 2023-10-05T21:08:24.005 INFO:teuthology.orchestra.run.smithi070.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2023-10-05T21:08:24.005 INFO:teuthology.orchestra.run.smithi070.stdout:writing requirements to s3tests.egg-info/requires.txt 2023-10-05T21:08:24.005 INFO:teuthology.orchestra.run.smithi070.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2023-10-05T21:08:24.006 INFO:teuthology.orchestra.run.smithi070.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T21:08:24.015 INFO:teuthology.orchestra.run.smithi070.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T21:08:24.016 INFO:teuthology.orchestra.run.smithi070.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T21:08:24.023 INFO:teuthology.orchestra.run.smithi070.stdout:running build_ext 2023-10-05T21:08:24.024 INFO:teuthology.orchestra.run.smithi070.stdout:Creating /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages/s3tests.egg-link (link to .) 2023-10-05T21:08:24.024 INFO:teuthology.orchestra.run.smithi070.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2023-10-05T21:08:24.024 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.024 INFO:teuthology.orchestra.run.smithi070.stdout:Installed /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T21:08:24.025 INFO:teuthology.orchestra.run.smithi070.stdout:Processing dependencies for s3tests==0.0.1 2023-10-05T21:08:24.132 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for isodate==0.6.1 2023-10-05T21:08:24.133 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: isodate 0.6.1 2023-10-05T21:08:24.133 INFO:teuthology.orchestra.run.smithi070.stdout:Adding isodate 0.6.1 to easy-install.pth file 2023-10-05T21:08:24.133 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.133 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.134 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for gevent==22.10.2 2023-10-05T21:08:24.134 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: gevent 22.10.2 2023-10-05T21:08:24.134 INFO:teuthology.orchestra.run.smithi070.stdout:Adding gevent 22.10.2 to easy-install.pth file 2023-10-05T21:08:24.134 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.134 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.134 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for munch==4.0.0 2023-10-05T21:08:24.134 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: munch 4.0.0 2023-10-05T21:08:24.135 INFO:teuthology.orchestra.run.smithi070.stdout:Adding munch 4.0.0 to easy-install.pth file 2023-10-05T21:08:24.135 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.135 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.136 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for PyYAML==6.0.1 2023-10-05T21:08:24.136 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: PyYAML 6.0.1 2023-10-05T21:08:24.136 INFO:teuthology.orchestra.run.smithi070.stdout:Adding PyYAML 6.0.1 to easy-install.pth file 2023-10-05T21:08:24.136 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.136 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.136 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for boto3==1.23.10 2023-10-05T21:08:24.136 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: boto3 1.23.10 2023-10-05T21:08:24.136 INFO:teuthology.orchestra.run.smithi070.stdout:Adding boto3 1.23.10 to easy-install.pth file 2023-10-05T21:08:24.136 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.136 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.137 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for boto==2.49.0 2023-10-05T21:08:24.137 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: boto 2.49.0 2023-10-05T21:08:24.138 INFO:teuthology.orchestra.run.smithi070.stdout:Adding boto 2.49.0 to easy-install.pth file 2023-10-05T21:08:24.138 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.138 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.138 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for six==1.16.0 2023-10-05T21:08:24.138 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: six 1.16.0 2023-10-05T21:08:24.138 INFO:teuthology.orchestra.run.smithi070.stdout:Adding six 1.16.0 to easy-install.pth file 2023-10-05T21:08:24.138 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.138 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.139 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for zope.interface==5.5.2 2023-10-05T21:08:24.140 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: zope.interface 5.5.2 2023-10-05T21:08:24.140 INFO:teuthology.orchestra.run.smithi070.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2023-10-05T21:08:24.140 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.140 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.140 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for greenlet==2.0.2 2023-10-05T21:08:24.140 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: greenlet 2.0.2 2023-10-05T21:08:24.140 INFO:teuthology.orchestra.run.smithi070.stdout:Adding greenlet 2.0.2 to easy-install.pth file 2023-10-05T21:08:24.140 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.140 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.140 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for zope.event==4.6 2023-10-05T21:08:24.140 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: zope.event 4.6 2023-10-05T21:08:24.140 INFO:teuthology.orchestra.run.smithi070.stdout:Adding zope.event 4.6 to easy-install.pth file 2023-10-05T21:08:24.142 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.142 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.142 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for setuptools==44.1.0 2023-10-05T21:08:24.142 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: setuptools 44.1.0 2023-10-05T21:08:24.142 INFO:teuthology.orchestra.run.smithi070.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2023-10-05T21:08:24.142 INFO:teuthology.orchestra.run.smithi070.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin 2023-10-05T21:08:24.142 INFO:teuthology.orchestra.run.smithi070.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin 2023-10-05T21:08:24.143 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.143 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.148 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for importlib-metadata==4.8.3 2023-10-05T21:08:24.148 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: importlib-metadata 4.8.3 2023-10-05T21:08:24.148 INFO:teuthology.orchestra.run.smithi070.stdout:Adding importlib-metadata 4.8.3 to easy-install.pth file 2023-10-05T21:08:24.148 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.148 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.148 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for s3transfer==0.5.2 2023-10-05T21:08:24.148 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: s3transfer 0.5.2 2023-10-05T21:08:24.149 INFO:teuthology.orchestra.run.smithi070.stdout:Adding s3transfer 0.5.2 to easy-install.pth file 2023-10-05T21:08:24.149 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.149 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.149 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for botocore==1.26.10 2023-10-05T21:08:24.149 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: botocore 1.26.10 2023-10-05T21:08:24.149 INFO:teuthology.orchestra.run.smithi070.stdout:Adding botocore 1.26.10 to easy-install.pth file 2023-10-05T21:08:24.150 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.150 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.150 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for jmespath==0.10.0 2023-10-05T21:08:24.150 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: jmespath 0.10.0 2023-10-05T21:08:24.150 INFO:teuthology.orchestra.run.smithi070.stdout:Adding jmespath 0.10.0 to easy-install.pth file 2023-10-05T21:08:24.151 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.151 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.151 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for typing-extensions==4.1.1 2023-10-05T21:08:24.151 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: typing-extensions 4.1.1 2023-10-05T21:08:24.151 INFO:teuthology.orchestra.run.smithi070.stdout:Adding typing-extensions 4.1.1 to easy-install.pth file 2023-10-05T21:08:24.151 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.152 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.152 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for zipp==3.6.0 2023-10-05T21:08:24.152 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: zipp 3.6.0 2023-10-05T21:08:24.152 INFO:teuthology.orchestra.run.smithi070.stdout:Adding zipp 3.6.0 to easy-install.pth file 2023-10-05T21:08:24.152 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.152 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.152 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for python-dateutil==2.8.2 2023-10-05T21:08:24.152 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: python-dateutil 2.8.2 2023-10-05T21:08:24.152 INFO:teuthology.orchestra.run.smithi070.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2023-10-05T21:08:24.152 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.153 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.153 INFO:teuthology.orchestra.run.smithi070.stdout:Searching for urllib3==1.26.17 2023-10-05T21:08:24.153 INFO:teuthology.orchestra.run.smithi070.stdout:Best match: urllib3 1.26.17 2023-10-05T21:08:24.153 INFO:teuthology.orchestra.run.smithi070.stdout:Adding urllib3 1.26.17 to easy-install.pth file 2023-10-05T21:08:24.154 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:08:24.154 INFO:teuthology.orchestra.run.smithi070.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T21:08:24.154 INFO:teuthology.orchestra.run.smithi070.stdout:Finished processing dependencies for s3tests==0.0.1 2023-10-05T21:08:24.167 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:08:24.167 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2023-10-05T21:08:24.185 INFO:tasks.s3tests:Configuring boto... 2023-10-05T21:08:24.186 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-10-05T21:08:24.186 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/boto-client.0.cfg 2023-10-05T21:08:24.243 DEBUG:teuthology.orchestra.run.smithi070:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain' 2023-10-05T21:08:24.706 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2023-10-05T21:08:24.757 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2023-10-05T21:08:24.808 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2023-10-05T21:08:24.858 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2023-10-05T21:08:24.908 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2023-10-05T21:08:24.916 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2023-10-05T21:08:24.973 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2023-10-05T21:08:24.981 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2023-10-05T21:08:25.023 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2023-10-05T21:08:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2023-10-05T21:08:25.116 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2023-10-05T21:08:25.174 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2023-10-05T21:08:25.224 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2023-10-05T21:08:25.276 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2023-10-05T21:08:25.277 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2023-10-05T21:08:25.320 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2023-10-05T21:08:25.320 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2023-10-05T21:08:25.320 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2023-10-05T21:08:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2023-10-05T21:08:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2023-10-05T21:08:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2023-10-05T21:08:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2023-10-05T21:08:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2023-10-05T21:08:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2023-10-05T21:08:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2023-10-05T21:08:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2023-10-05T21:08:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2023-10-05T21:08:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2023-10-05T21:08:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2023-10-05T21:08:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2023-10-05T21:08:25.322 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2023-10-05T21:08:25.322 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2023-10-05T21:08:25.322 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2023-10-05T21:08:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-10-05T21:08:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2023-10-05T21:08:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2023-10-05T21:08:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2023-10-05T21:08:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2023-10-05T21:08:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2023-10-05T21:08:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2023-10-05T21:08:25.326 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-10-05T21:08:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2023-10-05T21:08:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2023-10-05T21:08:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2023-10-05T21:08:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2023-10-05T21:08:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2023-10-05T21:08:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2023-10-05T21:08:25.464 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2023-10-05T21:08:25.555 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2023-10-05T21:08:25.593 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2023-10-05T21:08:27.488 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2023-10-05T21:08:27.583 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2023-10-05T21:08:27.651 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2023-10-05T21:08:27.669 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2023-10-05T21:08:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2023-10-05T21:08:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2023-10-05T21:08:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2023-10-05T21:08:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2023-10-05T21:08:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2023-10-05T21:08:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2023-10-05T21:08:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2023-10-05T21:08:27.747 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2023-10-05T21:08:27.747 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2023-10-05T21:08:27.747 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2023-10-05T21:08:27.747 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2023-10-05T21:08:27.747 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2023-10-05T21:08:27.747 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2023-10-05T21:08:27.747 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_pubilc_errordoc ... SKIP 2023-10-05T21:08:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2023-10-05T21:08:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2023-10-05T21:08:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2023-10-05T21:08:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2023-10-05T21:08:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2023-10-05T21:08:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2023-10-05T21:08:27.749 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2023-10-05T21:08:27.749 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2023-10-05T21:08:27.749 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2023-10-05T21:08:27.749 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2023-10-05T21:08:27.749 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2023-10-05T21:08:27.749 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2023-10-05T21:08:27.749 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2023-10-05T21:08:27.750 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2023-10-05T21:08:27.750 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2023-10-05T21:08:27.750 INFO:teuthology.orchestra.run.smithi070.stderr:SKIP 2023-10-05T21:08:29.324 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests.functional.test_utils.test_generate ... ok 2023-10-05T21:08:29.324 INFO:teuthology.orchestra.run.smithi070.stderr:/home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2023-10-05T21:08:29.324 INFO:teuthology.orchestra.run.smithi070.stderr: warnings.warn(warning, PythonDeprecationWarning) 2023-10-05T21:08:29.349 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2023-10-05T21:08:37.312 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2023-10-05T21:08:37.344 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2023-10-05T21:08:37.386 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2023-10-05T21:08:37.426 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2023-10-05T21:08:37.464 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2023-10-05T21:08:37.525 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2023-10-05T21:08:37.547 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2023-10-05T21:08:37.584 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2023-10-05T21:08:37.609 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2023-10-05T21:08:37.636 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2023-10-05T21:08:37.659 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2023-10-05T21:08:37.673 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2023-10-05T21:08:37.688 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2023-10-05T21:08:37.696 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2023-10-05T21:08:37.721 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2023-10-05T21:08:37.751 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2023-10-05T21:08:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2023-10-05T21:08:37.806 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2023-10-05T21:08:37.828 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2023-10-05T21:08:37.853 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2023-10-05T21:08:37.876 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2023-10-05T21:08:37.900 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2023-10-05T21:08:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2023-10-05T21:08:37.928 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2023-10-05T21:08:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2023-10-05T21:08:37.946 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2023-10-05T21:08:37.954 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2023-10-05T21:08:37.962 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2023-10-05T21:08:37.970 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2023-10-05T21:08:38.050 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2023-10-05T21:08:38.137 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2023-10-05T21:08:38.229 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2023-10-05T21:08:38.320 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2023-10-05T21:08:38.406 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2023-10-05T21:08:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2023-10-05T21:08:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2023-10-05T21:08:38.589 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2023-10-05T21:08:38.642 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2023-10-05T21:08:38.763 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2023-10-05T21:08:38.883 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2023-10-05T21:08:38.917 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2023-10-05T21:08:38.950 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2023-10-05T21:08:39.003 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2023-10-05T21:08:39.056 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2023-10-05T21:08:39.200 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2023-10-05T21:08:39.321 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2023-10-05T21:08:39.377 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2023-10-05T21:08:39.429 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2023-10-05T21:08:39.482 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2023-10-05T21:08:39.536 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2023-10-05T21:08:39.589 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2023-10-05T21:08:39.641 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2023-10-05T21:08:39.694 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2023-10-05T21:08:39.748 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2023-10-05T21:08:39.802 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2023-10-05T21:08:39.854 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2023-10-05T21:08:39.907 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2023-10-05T21:08:39.985 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2023-10-05T21:08:40.031 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2023-10-05T21:08:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2023-10-05T21:08:40.126 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2023-10-05T21:08:40.180 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2023-10-05T21:08:40.232 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2023-10-05T21:08:46.867 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2023-10-05T21:08:46.913 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2023-10-05T21:08:46.985 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2023-10-05T21:08:47.033 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2023-10-05T21:08:47.080 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2023-10-05T21:08:47.127 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2023-10-05T21:08:47.174 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2023-10-05T21:08:47.219 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2023-10-05T21:08:47.268 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2023-10-05T21:08:47.315 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2023-10-05T21:08:47.365 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2023-10-05T21:08:47.412 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2023-10-05T21:08:47.459 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2023-10-05T21:08:47.512 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2023-10-05T21:08:47.566 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2023-10-05T21:08:47.619 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2023-10-05T21:08:47.672 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2023-10-05T21:08:47.725 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2023-10-05T21:08:47.778 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2023-10-05T21:08:47.857 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2023-10-05T21:08:47.910 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2023-10-05T21:08:47.962 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-10-05T21:08:48.016 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-10-05T21:08:48.114 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2023-10-05T21:08:48.213 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2023-10-05T21:08:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2023-10-05T21:08:48.319 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2023-10-05T21:08:48.373 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2023-10-05T21:08:48.428 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2023-10-05T21:08:48.646 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_account_usage ... ok 2023-10-05T21:08:48.678 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_head_bucket_usage ... ok 2023-10-05T21:08:49.063 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2023-10-05T21:08:49.466 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2023-10-05T21:08:49.518 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2023-10-05T21:08:49.572 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2023-10-05T21:08:49.627 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2023-10-05T21:08:49.681 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2023-10-05T21:08:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken ... ok 2023-10-05T21:08:49.876 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_both_continuationtoken_startafter ... ok 2023-10-05T21:08:49.930 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2023-10-05T21:08:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2023-10-05T21:08:50.039 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2023-10-05T21:08:50.092 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2023-10-05T21:08:50.146 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2023-10-05T21:08:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2023-10-05T21:08:50.428 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2023-10-05T21:08:50.760 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2023-10-05T21:08:50.793 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2023-10-05T21:08:50.826 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2023-10-05T21:08:50.850 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2023-10-05T21:08:50.873 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2023-10-05T21:08:50.883 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2023-10-05T21:08:50.892 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2023-10-05T21:08:50.900 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2023-10-05T21:08:50.941 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2023-10-05T21:08:51.094 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2023-10-05T21:08:51.103 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2023-10-05T21:08:51.196 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2023-10-05T21:08:51.221 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_read_not_exist ... ok 2023-10-05T21:08:51.247 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2023-10-05T21:08:51.469 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2023-10-05T21:08:51.690 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2023-10-05T21:08:58.679 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete_key_limit ... ok 2023-10-05T21:09:05.646 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete_key_limit ... ok 2023-10-05T21:09:05.677 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2023-10-05T21:09:05.705 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2023-10-05T21:09:05.736 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2023-10-05T21:09:05.768 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2023-10-05T21:09:05.896 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2023-10-05T21:09:05.928 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2023-10-05T21:09:05.961 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2023-10-05T21:09:06.010 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2023-10-05T21:09:06.049 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2023-10-05T21:09:06.120 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2023-10-05T21:09:06.187 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2023-10-05T21:09:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2023-10-05T21:09:06.287 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2023-10-05T21:09:06.306 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2023-10-05T21:09:06.327 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2023-10-05T21:09:06.350 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2023-10-05T21:09:06.406 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2023-10-05T21:09:06.448 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2023-10-05T21:09:06.475 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2023-10-05T21:09:06.501 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2023-10-05T21:09:06.532 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2023-10-05T21:09:06.602 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2023-10-05T21:09:06.652 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2023-10-05T21:09:06.674 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2023-10-05T21:09:06.696 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2023-10-05T21:09:06.719 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2023-10-05T21:09:06.741 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2023-10-05T21:09:06.763 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2023-10-05T21:09:06.796 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2023-10-05T21:09:06.818 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2023-10-05T21:09:06.840 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2023-10-05T21:09:06.864 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2023-10-05T21:09:06.886 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2023-10-05T21:09:06.908 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2023-10-05T21:09:06.932 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2023-10-05T21:09:06.953 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2023-10-05T21:09:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2023-10-05T21:09:06.999 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2023-10-05T21:09:07.023 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2023-10-05T21:09:07.045 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2023-10-05T21:09:07.069 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2023-10-05T21:09:07.141 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2023-10-05T21:09:07.216 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2023-10-05T21:09:07.250 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2023-10-05T21:09:07.322 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2023-10-05T21:09:07.396 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2023-10-05T21:09:08.438 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2023-10-05T21:09:08.522 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2023-10-05T21:09:08.599 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2023-10-05T21:09:08.723 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2023-10-05T21:09:08.887 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2023-10-05T21:09:09.009 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2023-10-05T21:09:09.121 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2023-10-05T21:09:09.243 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2023-10-05T21:09:09.405 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2023-10-05T21:09:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2023-10-05T21:09:09.664 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2023-10-05T21:09:09.697 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2023-10-05T21:09:09.767 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2023-10-05T21:09:09.838 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2023-10-05T21:09:09.880 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2023-10-05T21:09:09.904 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2023-10-05T21:09:09.912 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_notexist ... ok 2023-10-05T21:09:09.974 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2023-10-05T21:09:10.006 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2023-10-05T21:09:10.037 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2023-10-05T21:09:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2023-10-05T21:09:10.103 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2023-10-05T21:09:10.133 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2023-10-05T21:09:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2023-10-05T21:09:10.271 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2023-10-05T21:09:10.351 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2023-10-05T21:09:10.383 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2023-10-05T21:09:10.413 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2023-10-05T21:09:10.444 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2023-10-05T21:09:10.474 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2023-10-05T21:09:10.511 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2023-10-05T21:09:10.549 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2023-10-05T21:09:10.605 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2023-10-05T21:09:10.635 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2023-10-05T21:09:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2023-10-05T21:09:10.657 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2023-10-05T21:09:10.667 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2023-10-05T21:09:10.677 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2023-10-05T21:09:10.699 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2023-10-05T21:09:10.759 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2023-10-05T21:09:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2023-10-05T21:09:10.849 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl ... ok 2023-10-05T21:09:10.912 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl ... ok 2023-10-05T21:09:11.047 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2023-10-05T21:09:11.047 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2023-10-05T21:09:11.106 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2023-10-05T21:09:11.166 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2023-10-05T21:09:11.226 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2023-10-05T21:09:11.299 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2023-10-05T21:09:11.372 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2023-10-05T21:09:11.494 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2023-10-05T21:09:11.594 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2023-10-05T21:09:11.639 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2023-10-05T21:09:11.761 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2023-10-05T21:09:11.883 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2023-10-05T21:09:11.968 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2023-10-05T21:09:12.018 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2023-10-05T21:09:12.043 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2023-10-05T21:09:12.166 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2023-10-05T21:09:12.287 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2023-10-05T21:09:12.409 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2023-10-05T21:09:12.530 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2023-10-05T21:09:12.653 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2023-10-05T21:09:12.773 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2023-10-05T21:09:12.909 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2023-10-05T21:09:13.021 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2023-10-05T21:09:13.134 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2023-10-05T21:09:13.245 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2023-10-05T21:09:13.279 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2023-10-05T21:09:13.387 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2023-10-05T21:09:13.460 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2023-10-05T21:09:13.538 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2023-10-05T21:09:13.616 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2023-10-05T21:09:13.650 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_not_exist ... ok 2023-10-05T21:09:13.775 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2023-10-05T21:09:13.978 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2023-10-05T21:09:14.211 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2023-10-05T21:09:14.416 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2023-10-05T21:09:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2023-10-05T21:09:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2023-10-05T21:09:15.030 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2023-10-05T21:09:15.237 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2023-10-05T21:09:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2023-10-05T21:09:15.653 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2023-10-05T21:09:15.816 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2023-10-05T21:09:15.980 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2023-10-05T21:09:16.175 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2023-10-05T21:09:16.344 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2023-10-05T21:09:16.479 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_buckets_list_ctime ... ok 2023-10-05T21:09:16.486 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2023-10-05T21:09:16.495 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2023-10-05T21:09:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2023-10-05T21:09:16.662 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2023-10-05T21:09:16.822 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2023-10-05T21:09:16.839 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2023-10-05T21:09:16.855 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2023-10-05T21:09:16.918 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2023-10-05T21:09:17.576 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2023-10-05T21:09:17.650 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2023-10-05T21:09:17.748 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2023-10-05T21:09:17.875 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2023-10-05T21:09:18.001 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2023-10-05T21:09:18.083 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2023-10-05T21:09:18.162 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2023-10-05T21:09:18.331 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2023-10-05T21:09:18.376 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2023-10-05T21:09:18.492 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2023-10-05T21:09:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2023-10-05T21:09:18.847 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2023-10-05T21:09:19.029 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2023-10-05T21:09:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2023-10-05T21:09:19.084 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2023-10-05T21:09:19.652 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2023-10-05T21:09:19.799 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2023-10-05T21:09:20.809 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2023-10-05T21:09:20.847 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2023-10-05T21:09:20.907 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2023-10-05T21:09:21.173 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2023-10-05T21:09:21.286 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2023-10-05T21:09:21.505 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2023-10-05T21:09:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2023-10-05T21:09:23.167 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2023-10-05T21:09:24.630 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2023-10-05T21:09:28.708 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2023-10-05T21:09:29.486 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2023-10-05T21:09:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2023-10-05T21:09:30.952 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2023-10-05T21:09:31.279 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2023-10-05T21:09:31.602 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2023-10-05T21:09:31.768 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2023-10-05T21:09:31.843 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2023-10-05T21:09:32.160 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2023-10-05T21:09:32.304 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload_owner ... ok 2023-10-05T21:09:32.390 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2023-10-05T21:09:32.475 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2023-10-05T21:09:32.499 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2023-10-05T21:09:32.622 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2023-10-05T21:09:35.736 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2023-10-05T21:09:38.782 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2023-10-05T21:09:41.825 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2023-10-05T21:09:41.958 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_set_bucket_tagging ... ok 2023-10-05T21:09:42.059 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2023-10-05T21:09:42.228 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2023-10-05T21:09:42.495 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2023-10-05T21:09:42.634 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2023-10-05T21:09:42.862 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2023-10-05T21:09:43.254 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2023-10-05T21:09:43.375 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2023-10-05T21:09:43.625 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2023-10-05T21:09:44.046 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2023-10-05T21:09:44.170 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2023-10-05T21:09:44.339 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2023-10-05T21:09:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2023-10-05T21:09:44.676 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2023-10-05T21:09:50.369 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2023-10-05T21:09:50.443 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2023-10-05T21:09:50.518 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2023-10-05T21:09:50.593 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2023-10-05T21:09:50.669 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2023-10-05T21:09:50.904 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2023-10-05T21:09:56.773 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2023-10-05T21:09:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2023-10-05T21:09:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2023-10-05T21:09:57.550 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2023-10-05T21:09:57.887 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2023-10-05T21:10:00.202 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2023-10-05T21:10:03.323 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2023-10-05T21:10:15.565 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2023-10-05T21:10:16.950 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2023-10-05T21:10:17.605 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2023-10-05T21:10:18.376 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2023-10-05T21:10:18.634 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2023-10-05T21:10:18.912 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2023-10-05T21:10:19.041 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2023-10-05T21:10:19.296 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2023-10-05T21:10:19.493 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2023-10-05T21:10:19.794 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... ok 2023-10-05T21:10:19.825 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2023-10-05T21:10:19.901 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2023-10-05T21:10:19.977 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2023-10-05T21:10:20.003 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2023-10-05T21:10:20.030 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2023-10-05T21:10:20.147 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2023-10-05T21:10:20.177 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2023-10-05T21:10:20.205 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2023-10-05T21:10:20.253 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2023-10-05T21:10:20.283 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2023-10-05T21:10:20.314 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2023-10-05T21:10:20.344 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2023-10-05T21:10:20.375 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2023-10-05T21:10:20.405 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_set_invalid_date ... ok 2023-10-05T21:10:20.405 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition ... SKIP 2023-10-05T21:10:20.405 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2023-10-05T21:10:20.406 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2023-10-05T21:10:20.406 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition ... SKIP 2023-10-05T21:10:20.406 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_multiple_transition ... SKIP 2023-10-05T21:10:20.406 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition_large_obj ... SKIP 2023-10-05T21:10:20.479 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2023-10-05T21:10:20.556 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2023-10-05T21:10:20.608 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2023-10-05T21:10:20.682 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2023-10-05T21:10:20.726 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2023-10-05T21:10:20.802 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2023-10-05T21:10:20.878 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2023-10-05T21:10:20.945 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2023-10-05T21:10:21.014 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2023-10-05T21:10:21.082 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2023-10-05T21:10:21.151 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2023-10-05T21:10:22.026 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2023-10-05T21:10:22.574 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2023-10-05T21:10:22.611 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2023-10-05T21:10:22.656 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2023-10-05T21:10:22.733 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2023-10-05T21:10:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2023-10-05T21:10:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2023-10-05T21:10:23.689 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2023-10-05T21:10:24.159 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2023-10-05T21:10:24.627 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2023-10-05T21:10:24.663 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2023-10-05T21:10:24.739 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2023-10-05T21:10:24.815 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2023-10-05T21:10:24.866 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2023-10-05T21:10:24.941 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2023-10-05T21:10:25.018 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2023-10-05T21:10:25.062 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2023-10-05T21:10:25.108 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2023-10-05T21:10:25.166 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2023-10-05T21:10:25.224 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2023-10-05T21:10:25.352 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2023-10-05T21:10:25.506 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2023-10-05T21:10:25.674 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2023-10-05T21:10:25.794 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2023-10-05T21:10:25.956 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2023-10-05T21:10:26.113 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2023-10-05T21:10:26.282 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2023-10-05T21:10:26.444 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2023-10-05T21:10:26.607 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2023-10-05T21:10:26.815 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2023-10-05T21:10:27.022 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2023-10-05T21:10:27.135 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2023-10-05T21:10:27.170 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2023-10-05T21:10:27.288 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2023-10-05T21:10:27.427 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2023-10-05T21:10:27.599 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2023-10-05T21:10:27.774 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2023-10-05T21:10:28.008 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2023-10-05T21:10:29.583 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2023-10-05T21:10:29.732 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2023-10-05T21:10:29.942 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2023-10-05T21:10:30.114 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2023-10-05T21:10:30.383 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2023-10-05T21:10:30.554 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2023-10-05T21:10:30.641 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2023-10-05T21:10:30.802 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2023-10-05T21:10:30.830 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_s3 ... ok 2023-10-05T21:10:30.857 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_kms ... ok 2023-10-05T21:10:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_s3_kms ... ok 2023-10-05T21:10:30.916 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_bad_enc_kms ... ok 2023-10-05T21:10:30.999 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_s3_noenc ... ok 2023-10-05T21:10:31.072 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_s3_kms ... ok 2023-10-05T21:10:31.152 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_noenc ... ok 2023-10-05T21:10:31.225 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_s3 ... ok 2023-10-05T21:10:31.467 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2023-10-05T21:10:31.540 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2023-10-05T21:10:31.604 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2023-10-05T21:10:31.669 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2023-10-05T21:10:31.736 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2023-10-05T21:10:31.800 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2023-10-05T21:10:31.907 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_mode ... ok 2023-10-05T21:10:31.974 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2023-10-05T21:10:32.038 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2023-10-05T21:10:32.107 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2023-10-05T21:10:32.171 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2023-10-05T21:10:32.219 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2023-10-05T21:10:32.294 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2023-10-05T21:10:32.414 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2023-10-05T21:10:32.509 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2023-10-05T21:10:32.567 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_iso8601 ... ok 2023-10-05T21:10:32.642 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2023-10-05T21:10:32.745 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2023-10-05T21:10:32.847 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2023-10-05T21:10:32.949 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2023-10-05T21:10:33.044 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2023-10-05T21:10:33.144 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2023-10-05T21:10:33.238 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2023-10-05T21:10:33.393 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention_and_marker ... ok 2023-10-05T21:10:33.531 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_multi_delete_object_with_retention ... ok 2023-10-05T21:10:33.575 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2023-10-05T21:10:33.647 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2023-10-05T21:10:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2023-10-05T21:10:33.856 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2023-10-05T21:10:33.928 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2023-10-05T21:10:34.017 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2023-10-05T21:10:34.068 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2023-10-05T21:10:34.136 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2023-10-05T21:10:34.192 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2023-10-05T21:10:34.230 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_with_bypass ... ok 2023-10-05T21:10:34.309 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_without_bypass ... ok 2023-10-05T21:10:34.386 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_compliance ... ok 2023-10-05T21:10:34.508 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifmatch_good ... ok 2023-10-05T21:10:34.632 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifnonematch_failed ... ok 2023-10-05T21:10:34.660 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2023-10-05T21:10:34.764 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2023-10-05T21:10:34.839 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2023-10-05T21:10:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2023-10-05T21:10:34.993 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2023-10-05T21:10:35.068 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2023-10-05T21:10:35.096 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2023-10-05T21:10:35.168 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2023-10-05T21:10:35.373 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2023-10-05T21:10:35.536 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2023-10-05T21:10:35.610 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2023-10-05T21:10:35.797 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2023-10-05T21:10:36.568 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_on_a_bucket_with_policy ... ok 2023-10-05T21:10:36.597 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_bucket_encryption_s3 ... ok 2023-10-05T21:10:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_put_bucket_encryption_kms ... ok 2023-10-05T21:10:36.704 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_encryption_s3 ... ok 2023-10-05T21:10:36.781 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_encryption_kms ... ok 2023-10-05T21:10:36.864 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_delete_bucket_encryption_s3 ... ok 2023-10-05T21:10:36.947 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_delete_bucket_encryption_kms ... ok 2023-10-05T21:10:37.032 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_upload_1b ... ok 2023-10-05T21:10:37.116 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_upload_1kb ... ok 2023-10-05T21:10:37.176 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_upload_1mb ... ok 2023-10-05T21:10:37.334 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_upload_8mb ... ok 2023-10-05T21:10:37.418 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_upload_1b ... ok 2023-10-05T21:10:37.500 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_upload_1kb ... ok 2023-10-05T21:10:37.556 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_upload_1mb ... ok 2023-10-05T21:10:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_upload_8mb ... ok 2023-10-05T21:10:37.767 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_method_head ... ok 2023-10-05T21:10:38.581 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_multipart_upload ... ok 2023-10-05T21:10:38.678 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_default_post_object_authenticated_request ... ok 2023-10-05T21:10:38.770 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_post_object_authenticated_request ... ok 2023-10-05T21:10:38.858 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_encrypted_upload_1b ... ok 2023-10-05T21:10:38.978 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_encrypted_upload_1kb ... ok 2023-10-05T21:10:39.044 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_encrypted_upload_1mb ... ok 2023-10-05T21:10:39.228 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3.test_sse_s3_encrypted_upload_8mb ... ok 2023-10-05T21:10:40.200 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_generate_where_clause ... ok 2023-10-05T21:10:41.129 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_generate_projection ... ok 2023-10-05T21:10:41.182 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_count_operation ... ok 2023-10-05T21:10:41.590 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_column_sum_min_max ... ok 2023-10-05T21:10:42.138 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_nullif_expressions ... ok 2023-10-05T21:10:42.382 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_nulliftrue_expressions ... ok 2023-10-05T21:10:42.595 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_is_not_null_expressions ... ok 2023-10-05T21:10:42.668 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_lowerupper_expressions ... ok 2023-10-05T21:10:43.205 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_in_expressions ... ok 2023-10-05T21:10:43.566 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_true_false_in_expressions ... ok 2023-10-05T21:10:44.076 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_like_expressions ... ok 2023-10-05T21:10:44.415 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_truefalselike_expressions ... ok 2023-10-05T21:10:44.663 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_complex_expressions ... ok 2023-10-05T21:10:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_alias ... ok 2023-10-05T21:10:45.011 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_alias_cyclic_refernce ... ok 2023-10-05T21:10:45.212 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_datetime ... ok 2023-10-05T21:10:45.479 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_true_false_datetime ... ok 2023-10-05T21:10:45.570 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_csv_parser ... ok 2023-10-05T21:10:45.784 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_csv_definition ... ok 2023-10-05T21:10:45.994 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_schema_definition ... ok 2023-10-05T21:10:46.248 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_when_then_else_expressions ... ok 2023-10-05T21:10:46.481 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_coalesce_expressions ... ok 2023-10-05T21:10:46.691 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_cast_expressions ... ok 2023-10-05T21:10:46.692 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_version ... ok 2023-10-05T21:10:47.054 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_trim_expressions ... ok 2023-10-05T21:10:47.437 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_truefalse_trim_expressions ... ok 2023-10-05T21:10:47.688 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_escape_expressions ... ok 2023-10-05T21:10:47.879 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_case_value_expressions ... ok 2023-10-05T21:10:48.058 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_bool_cast_expressions ... ok 2023-10-05T21:11:00.920 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_progress_expressions ... ok 2023-10-05T21:11:00.920 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_s3select.test_output_serial_expressions ... ok 2023-10-05T21:11:58.210 INFO:teuthology.orchestra.run.smithi070.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2023-10-05T21:11:58.679 INFO:teuthology.orchestra.run.smithi070.stderr: 2023-10-05T21:11:58.680 INFO:teuthology.orchestra.run.smithi070.stderr:---------------------------------------------------------------------- 2023-10-05T21:11:58.680 INFO:teuthology.orchestra.run.smithi070.stderr:Ran 638 tests in 213.753s 2023-10-05T21:11:58.680 INFO:teuthology.orchestra.run.smithi070.stderr: 2023-10-05T21:11:58.680 INFO:teuthology.orchestra.run.smithi070.stderr:OK (SKIP=79) 2023-10-05T21:11:58.681 INFO:teuthology.orchestra.run.smithi070.stderr:Exception ignored in: .generate at 0x7fb785357570> 2023-10-05T21:11:58.681 INFO:teuthology.orchestra.run.smithi070.stderr:RuntimeError: generator ignored GeneratorExit 2023-10-05T21:11:58.681 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2023-10-05T21:11:58.682 DEBUG:teuthology.orchestra.run.smithi070:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2023-10-05T21:11:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T21:11:58.752 INFO:tasks.s3tests:Cleaning up boto... 2023-10-05T21:11:58.753 DEBUG:teuthology.orchestra.run.smithi070:> rm /home/ubuntu/cephtest/boto-client.0.cfg 2023-10-05T21:11:58.770 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2023-10-05T21:11:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setuser ceph since I am not root 2023-10-05T21:11:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T21:11:58.860 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.858+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.861 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.859+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:58.861 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.859+0000 7fbf95347680 20 realm 2023-10-05T21:11:58.861 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.859+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.861 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.859+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:58.861 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.859+0000 7fbf95347680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T21:11:58.861 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.859+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.862 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.859+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:58.862 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.859+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.862 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.859+0000 7fbf95347680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:58.862 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.860+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.862 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.860+0000 7fbf95347680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:11:58.863 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.860+0000 7fbf95347680 20 searching for the correct realm 2023-10-05T21:11:58.869 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.867+0000 7fbf95347680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:11:58.870 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.867+0000 7fbf95347680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:11:58.870 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.867+0000 7fbf95347680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:11:58.870 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.867+0000 7fbf95347680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:11:58.870 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.867+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.870 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.867+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:58.870 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.867+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.870 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.868+0000 7fbf95347680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:58.870 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.868+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.868+0000 7fbf95347680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:11:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.868+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.868+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.868+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.868+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.868+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.872 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.868+0000 7fbf95347680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:58.877 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.874+0000 7fbf95347680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:11:58.877 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.874+0000 7fbf95347680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:11:58.877 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.874+0000 7fbf95347680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:11:58.877 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.874+0000 7fbf95347680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:11:58.877 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.874+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.877 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.875+0000 7fbf95347680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:58.877 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.875+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.875+0000 7fbf95347680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:11:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.875+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.875+0000 7fbf95347680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.875+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.875+0000 7fbf95347680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:11:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.875+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 20 zone default found 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 4 Realm: () 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 4 ZoneGroup: default (cb9d3d40-68a5-4791-bf47-2aada3a09714) 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 4 Zone: default (bb6df8cb-740d-4b21-924c-79673a5d256e) 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 20 zonegroup default 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 20 started sync module instance, tier type = 2023-10-05T21:11:58.880 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.876+0000 7fbf95347680 20 started zone id=bb6df8cb-740d-4b21-924c-79673a5d256e (name=default) with tier type = 2023-10-05T21:11:58.886 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.884+0000 7fbf95347680 20 add_watcher() i=0 2023-10-05T21:11:58.886 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.884+0000 7fbf95347680 20 add_watcher() i=1 2023-10-05T21:11:58.886 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.884+0000 7fbf95347680 20 add_watcher() i=2 2023-10-05T21:11:58.886 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.884+0000 7fbf95347680 20 add_watcher() i=3 2023-10-05T21:11:58.886 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.884+0000 7fbf95347680 20 add_watcher() i=4 2023-10-05T21:11:58.886 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.884+0000 7fbf95347680 20 add_watcher() i=5 2023-10-05T21:11:58.886 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.884+0000 7fbf95347680 20 add_watcher() i=6 2023-10-05T21:11:58.887 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.884+0000 7fbf95347680 20 add_watcher() i=7 2023-10-05T21:11:58.887 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.884+0000 7fbf95347680 2 all 8 watchers are set, enabling cache 2023-10-05T21:11:58.888 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.886+0000 7fbf59fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:11:58.888 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.886+0000 7fbf95347680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T21:11:58.888 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.886+0000 7fbf95347680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T21:11:58.926 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.923+0000 7fbed0ff9700 20 reqs_thread_entry: start 2023-10-05T21:11:58.926 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.923+0000 7fbec67fc700 20 reqs_thread_entry: start 2023-10-05T21:11:58.926 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.923+0000 7fbf95347680 20 init_complete bucket index max shards: 11 2023-10-05T21:11:58.926 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.924+0000 7fbf95347680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T21:11:58.926 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.924+0000 7fbec4ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 21:12:29 2023 2023-10-05T21:11:58.926 INFO:teuthology.orchestra.run.smithi070.stderr: 2023-10-05T21:11:58.926 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.924+0000 7fbeabfff700 20 reqs_thread_entry: start 2023-10-05T21:11:58.927 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.924+0000 7fbf95347680 20 get_system_obj_state: rctx=0x7fff67b07db8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x56502f222240 s->prefetch_data=0 2023-10-05T21:11:58.927 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.924+0000 7fbf95347680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T21:11:58.927 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.924+0000 7fbf95347680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-10-05T21:11:58.927 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.924+0000 7fbf95347680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T21:11:58.927 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.924+0000 7fbf95347680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:58.927 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.924+0000 7fbf95347680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T21:11:58.927 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.924+0000 7fbf95347680 20 Read xattr: user.rgw.idtag 2023-10-05T21:11:58.927 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.924+0000 7fbf95347680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T21:11:58.927 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.924+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.929 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.925+0000 7fbf95347680 20 rados_obj.operate() r=0 bl.length=469 2023-10-05T21:11:58.929 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.925+0000 7fbf95347680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-10-05T21:11:58.929 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.925+0000 7fbf95347680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T21:11:58.929 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.925+0000 7fbf95347680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:58.929 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.925+0000 7fbf95347680 20 get_system_obj_state: rctx=0x7fff67b07a70 obj=default.rgw.meta:root:.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 state=0x56502f222240 s->prefetch_data=0 2023-10-05T21:11:58.929 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.925+0000 7fbf95347680 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 : miss 2023-10-05T21:11:58.929 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 info.flags=0x16 2023-10-05T21:11:58.929 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 10 adding default.rgw.meta+root+.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 to cache LRU end 2023-10-05T21:11:58.929 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:58.929 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T21:11:58.929 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T21:11:58.930 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 20 Read xattr: user.rgw.acl 2023-10-05T21:11:58.930 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 20 Read xattr: user.rgw.idtag 2023-10-05T21:11:58.930 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 : type miss (requested=0x13, cached=0x16) 2023-10-05T21:11:58.930 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.930 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 20 rados_obj.operate() r=0 bl.length=364 2023-10-05T21:11:58.930 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 info.flags=0x13 2023-10-05T21:11:58.930 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 10 moving default.rgw.meta+root+.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 to cache LRU end 2023-10-05T21:11:58.930 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:58.930 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T21:11:58.930 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 2023-10-05T21:11:58.930 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :test[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T21:11:58.930 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.926+0000 7fbf95347680 10 cls_bucket_list_unordered: got 831FCC 2023-10-05T21:11:58.931 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.927+0000 7fbf95347680 10 cls_bucket_list_unordered: got 8FEF5C 2023-10-05T21:11:58.931 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.927+0000 7fbf95347680 10 cls_bucket_list_unordered: got B16F00 2023-10-05T21:11:58.932 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.927+0000 7fbf95347680 10 cls_bucket_list_unordered: got 273C00 2023-10-05T21:11:58.932 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.927+0000 7fbf95347680 10 cls_bucket_list_unordered: got 811233 2023-10-05T21:11:58.933 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.928+0000 7fbf95347680 10 cls_bucket_list_unordered: got 9E7149 2023-10-05T21:11:58.933 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.928+0000 7fbf95347680 10 cls_bucket_list_unordered: got C6BF54 2023-10-05T21:11:58.933 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.928+0000 7fbf95347680 10 cls_bucket_list_unordered: got 9B777A 2023-10-05T21:11:58.933 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.928+0000 7fbf95347680 10 cls_bucket_list_unordered: got AB9E67 2023-10-05T21:11:58.933 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.928+0000 7fbf95347680 10 cls_bucket_list_unordered: got E384FF 2023-10-05T21:11:58.933 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.928+0000 7fbf95347680 10 cls_bucket_list_unordered: got A1088A 2023-10-05T21:11:58.933 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.929+0000 7fbf95347680 10 cls_bucket_list_unordered: got 204279 2023-10-05T21:11:58.933 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.929+0000 7fbf95347680 10 cls_bucket_list_unordered: got 477880 2023-10-05T21:11:58.933 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.929+0000 7fbf95347680 10 cls_bucket_list_unordered: got 2EEAF6 2023-10-05T21:11:58.933 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.929+0000 7fbf95347680 10 cls_bucket_list_unordered: got 3190D1 2023-10-05T21:11:58.933 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.929+0000 7fbf95347680 10 cls_bucket_list_unordered: got 3AD487 2023-10-05T21:11:58.933 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.929+0000 7fbf95347680 10 cls_bucket_list_unordered: got 652C1F 2023-10-05T21:11:58.934 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.929+0000 7fbf95347680 10 cls_bucket_list_unordered: got 884BB2 2023-10-05T21:11:58.934 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.929+0000 7fbf95347680 10 cls_bucket_list_unordered: got AA8D26 2023-10-05T21:11:58.934 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.929+0000 7fbf95347680 10 cls_bucket_list_unordered: got CA709F 2023-10-05T21:11:58.934 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.929+0000 7fbf95347680 10 cls_bucket_list_unordered: got DAA22E 2023-10-05T21:11:58.934 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 10 cls_bucket_list_unordered: got 20DAD6 2023-10-05T21:11:58.934 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 10 cls_bucket_list_unordered: got 666619 2023-10-05T21:11:58.934 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 10 cls_bucket_list_unordered: got DA6EFA 2023-10-05T21:11:58.934 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 10 cls_bucket_list_unordered: got 01BBD6 2023-10-05T21:11:58.934 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 10 cls_bucket_list_unordered: got 062A5C 2023-10-05T21:11:58.934 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 10 cls_bucket_list_unordered: got 2D52AA 2023-10-05T21:11:58.934 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 10 cls_bucket_list_unordered: got 931CFA 2023-10-05T21:11:58.935 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 10 cls_bucket_list_unordered: got DBFBFB 2023-10-05T21:11:58.935 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:831FCC state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.935 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 10 manifest: total_size = 399126 2023-10-05T21:11:58.935 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.4180912906712534931 2023-10-05T21:11:58.935 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:831FCC state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.935 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.935 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.930+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.0 2023-10-05T21:11:58.938 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : miss 2023-10-05T21:11:58.938 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.939 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:58.939 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 10 cache put: name=default.rgw.log++bucket.sync-source-hints.test info.flags=0x0 2023-10-05T21:11:58.939 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 10 adding default.rgw.log++bucket.sync-source-hints.test to cache LRU end 2023-10-05T21:11:58.939 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : miss 2023-10-05T21:11:58.939 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:58.939 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:58.939 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 10 cache put: name=default.rgw.log++bucket.sync-target-hints.test info.flags=0x0 2023-10-05T21:11:58.939 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 10 adding default.rgw.log++bucket.sync-target-hints.test to cache LRU end 2023-10-05T21:11:58.939 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.939 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.939 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.939 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:8FEF5C state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.940 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 10 manifest: total_size = 399260 2023-10-05T21:11:58.940 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.18426554968827328284 2023-10-05T21:11:58.940 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:8FEF5C state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.940 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.940 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.933+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.1 2023-10-05T21:11:58.940 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.934+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=831FCC 2023-10-05T21:11:58.940 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.936+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.940 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.936+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.940 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.936+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.940 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.936+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.940 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.936+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.940 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.936+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:B16F00 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.941 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.936+0000 7fbf95347680 10 manifest: total_size = 399265 2023-10-05T21:11:58.941 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.936+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.11336216694613938854 2023-10-05T21:11:58.941 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.936+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:B16F00 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.941 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.936+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.941 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.936+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.1 2023-10-05T21:11:58.942 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.937+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=8FEF5C 2023-10-05T21:11:58.942 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.939+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.942 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.939+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.942 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.939+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.942 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.939+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.942 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.939+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.942 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.939+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:273C00 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.943 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.939+0000 7fbf95347680 10 manifest: total_size = 590082 2023-10-05T21:11:58.943 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.939+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.10381782649794373656 2023-10-05T21:11:58.943 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.939+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:273C00 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.943 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.939+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.943 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.939+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.2 2023-10-05T21:11:58.943 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.940+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=B16F00 2023-10-05T21:11:58.944 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.941+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.944 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.941+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.944 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.941+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.944 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.941+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.944 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.941+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.944 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.941+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:811233 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.944 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.941+0000 7fbf95347680 10 manifest: total_size = 180000 2023-10-05T21:11:58.944 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.941+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.611547468606564683 2023-10-05T21:11:58.944 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.941+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:811233 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.944 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.941+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.944 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.941+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.2 2023-10-05T21:11:58.945 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.942+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=273C00 2023-10-05T21:11:58.946 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.944+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.947 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.944+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.947 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.944+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.947 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.944+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.947 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.944+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.947 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.944+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:9E7149 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.947 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.944+0000 7fbf95347680 10 manifest: total_size = 399052 2023-10-05T21:11:58.947 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.944+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.15101392422341701361 2023-10-05T21:11:58.947 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.944+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:9E7149 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.947 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.944+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.947 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.944+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.3 2023-10-05T21:11:58.948 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.945+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=811233 2023-10-05T21:11:58.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.946+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.946+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.946+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.946+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.946+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.946+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:C6BF54 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.947+0000 7fbf95347680 10 manifest: total_size = 399099 2023-10-05T21:11:58.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.947+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.7220006578531448429 2023-10-05T21:11:58.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.947+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:C6BF54 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.949 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.947+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.950 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.947+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.3 2023-10-05T21:11:58.950 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.947+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=9E7149 2023-10-05T21:11:58.951 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.949+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.952 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.949+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.952 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.949+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.952 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.949+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.952 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.949+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.952 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.949+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:9B777A state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.952 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.949+0000 7fbf95347680 10 manifest: total_size = 399157 2023-10-05T21:11:58.952 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.949+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.12640996426540388099 2023-10-05T21:11:58.952 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.950+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:9B777A state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.952 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.950+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.952 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.950+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.4 2023-10-05T21:11:58.953 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.950+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=C6BF54 2023-10-05T21:11:58.954 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.952+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.954 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.952+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.954 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.952+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.954 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.952+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.954 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.952+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.954 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.952+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:AB9E67 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.954 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.952+0000 7fbf95347680 10 manifest: total_size = 40 2023-10-05T21:11:58.955 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.952+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.5726599318928813412 2023-10-05T21:11:58.955 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.952+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:AB9E67 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.955 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.952+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.955 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.952+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.4 2023-10-05T21:11:58.955 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.953+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=9B777A 2023-10-05T21:11:58.957 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.954+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.957 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.954+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.957 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.954+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.957 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.954+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.957 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.954+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.957 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.954+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:E384FF state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.957 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.955+0000 7fbf95347680 10 manifest: total_size = 399180 2023-10-05T21:11:58.957 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.955+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.7436096436492315539 2023-10-05T21:11:58.957 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.955+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:E384FF state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.957 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.955+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.958 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.955+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.4 2023-10-05T21:11:58.958 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.955+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=AB9E67 2023-10-05T21:11:58.959 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.957+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.959 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.957+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.960 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.957+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.960 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.957+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.960 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.957+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.960 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.957+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:A1088A state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.960 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.958+0000 7fbf95347680 10 manifest: total_size = 399090 2023-10-05T21:11:58.960 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.958+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.16414370789155598707 2023-10-05T21:11:58.960 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.958+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:A1088A state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.960 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.958+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.960 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.958+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.5 2023-10-05T21:11:58.961 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.958+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=E384FF 2023-10-05T21:11:58.962 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.960+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.962 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.960+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.962 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.960+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.962 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.960+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.962 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.960+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.962 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.960+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:204279 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.963 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.960+0000 7fbf95347680 10 manifest: total_size = 5 2023-10-05T21:11:58.963 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.960+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.15399090355562204925 2023-10-05T21:11:58.963 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.960+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:204279 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.963 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.960+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.963 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.960+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.6 2023-10-05T21:11:58.963 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.961+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=A1088A 2023-10-05T21:11:58.964 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.962+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.964 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.962+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.964 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.962+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.965 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.962+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.965 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.962+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.965 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.962+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:477880 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.965 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.962+0000 7fbf95347680 10 manifest: total_size = 399129 2023-10-05T21:11:58.965 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.962+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.174307477107115493 2023-10-05T21:11:58.965 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.962+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:477880 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.965 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.962+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.965 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.962+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.6 2023-10-05T21:11:58.965 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.963+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=204279 2023-10-05T21:11:58.967 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.965+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.967 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.965+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.967 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.965+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.967 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.965+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.967 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.965+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.967 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.965+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:2EEAF6 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.968 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.965+0000 7fbf95347680 10 manifest: total_size = 398854 2023-10-05T21:11:58.968 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.965+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.13430234445682888503 2023-10-05T21:11:58.968 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.965+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:2EEAF6 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.968 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.965+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.968 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.965+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.7 2023-10-05T21:11:58.968 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.966+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=477880 2023-10-05T21:11:58.969 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.967+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.969 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.967+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.969 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.967+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.969 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.967+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.970 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.967+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.970 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.967+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:3190D1 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.970 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.968+0000 7fbf95347680 10 manifest: total_size = 399138 2023-10-05T21:11:58.970 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.968+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.13547409097027226881 2023-10-05T21:11:58.971 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.968+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:3190D1 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.971 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.968+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.971 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.968+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.8 2023-10-05T21:11:58.971 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.968+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=2EEAF6 2023-10-05T21:11:58.972 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.970+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.972 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.970+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.972 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.970+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.972 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.970+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.972 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.970+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.972 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.970+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:3AD487 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.972 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.970+0000 7fbf95347680 10 manifest: total_size = 399136 2023-10-05T21:11:58.973 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.970+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.7845308149732683871 2023-10-05T21:11:58.973 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.970+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:3AD487 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.973 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.970+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.973 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.970+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.8 2023-10-05T21:11:58.973 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.971+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=3190D1 2023-10-05T21:11:58.975 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.972+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.975 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.972+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.975 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.972+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.975 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.972+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.975 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.972+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.975 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.972+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:652C1F state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.975 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.973+0000 7fbf95347680 10 manifest: total_size = 39911538 2023-10-05T21:11:58.975 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.973+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.14102769078160510892 2023-10-05T21:11:58.975 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.973+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:652C1F state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.975 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.973+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.975 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.973+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.8 2023-10-05T21:11:58.976 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.973+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=3AD487 2023-10-05T21:11:58.979 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.979 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.979 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.979 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.979 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T21:11:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=0 2023-10-05T21:11:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T21:11:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=0 2023-10-05T21:11:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T21:11:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): result: ofs=16777216 stripe_ofs=16777216 part_ofs=0 rule->part_size=0 2023-10-05T21:11:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T21:11:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): result: ofs=20971520 stripe_ofs=20971520 part_ofs=0 rule->part_size=0 2023-10-05T21:11:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T21:11:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): result: ofs=25165824 stripe_ofs=25165824 part_ofs=0 rule->part_size=0 2023-10-05T21:11:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T21:11:58.980 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): result: ofs=29360128 stripe_ofs=29360128 part_ofs=0 rule->part_size=0 2023-10-05T21:11:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T21:11:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): result: ofs=33554432 stripe_ofs=33554432 part_ofs=0 rule->part_size=0 2023-10-05T21:11:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T21:11:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): result: ofs=37748736 stripe_ofs=37748736 part_ofs=0 rule->part_size=0 2023-10-05T21:11:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T21:11:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 RGWObjManifest::operator++(): result: ofs=39911538 stripe_ofs=39911538 part_ofs=0 rule->part_size=0 2023-10-05T21:11:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - tag is: bb6df8cb-740d-4b21-924c-79673a5d256e.4204.14102769078160510892 2023-10-05T21:11:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-10-05T21:11:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538__shadow_.klHZYYqIvb437DSvDpeRMfPIoHvnm_a_1 2023-10-05T21:11:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 333 2023-10-05T21:11:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538__shadow_.klHZYYqIvb437DSvDpeRMfPIoHvnm_a_2 2023-10-05T21:11:58.981 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 565 2023-10-05T21:11:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538__shadow_.klHZYYqIvb437DSvDpeRMfPIoHvnm_a_3 2023-10-05T21:11:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 797 2023-10-05T21:11:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538__shadow_.klHZYYqIvb437DSvDpeRMfPIoHvnm_a_4 2023-10-05T21:11:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1029 2023-10-05T21:11:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538__shadow_.klHZYYqIvb437DSvDpeRMfPIoHvnm_a_5 2023-10-05T21:11:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1261 2023-10-05T21:11:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538__shadow_.klHZYYqIvb437DSvDpeRMfPIoHvnm_a_6 2023-10-05T21:11:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1493 2023-10-05T21:11:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538__shadow_.klHZYYqIvb437DSvDpeRMfPIoHvnm_a_7 2023-10-05T21:11:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1725 2023-10-05T21:11:58.982 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538__shadow_.klHZYYqIvb437DSvDpeRMfPIoHvnm_a_8 2023-10-05T21:11:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1957 2023-10-05T21:11:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538__shadow_.klHZYYqIvb437DSvDpeRMfPIoHvnm_a_9 2023-10-05T21:11:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 2189 2023-10-05T21:11:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.975+0000 7fbf95347680 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-10-05T21:11:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.976+0000 7fbf95347680 20 garbage collection: RGWGC::send_chain - on object name: gc.19tag is: bb6df8cb-740d-4b21-924c-79673a5d256e.4204.14102769078160510892 2023-10-05T21:11:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.976+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=652C1F 2023-10-05T21:11:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.977+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:884BB2 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.977+0000 7fbf95347680 10 manifest: total_size = 5 2023-10-05T21:11:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.977+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.16225106832540112343 2023-10-05T21:11:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.977+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:884BB2 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.977+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.983 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.977+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.8 2023-10-05T21:11:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.979+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.979+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.979+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.979+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.979+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.979+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:AA8D26 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.979+0000 7fbf95347680 10 manifest: total_size = 399135 2023-10-05T21:11:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.979+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.7162342482631586838 2023-10-05T21:11:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.979+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:AA8D26 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.979+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.979+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.8 2023-10-05T21:11:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.980+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=884BB2 2023-10-05T21:11:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.982+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.982+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.982+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.982+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.982+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.982+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:CA709F state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.986 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.982+0000 7fbf95347680 10 manifest: total_size = 575322 2023-10-05T21:11:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.982+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.6126916149717331542 2023-10-05T21:11:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.982+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:CA709F state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.982+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.982+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.8 2023-10-05T21:11:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.983+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=AA8D26 2023-10-05T21:11:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.984+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.984+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.984+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.984+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.984+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.984+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:DAA22E state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.985+0000 7fbf95347680 10 manifest: total_size = 49216 2023-10-05T21:11:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.985+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.3308175740591355449 2023-10-05T21:11:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.985+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:DAA22E state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.985+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.985+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.8 2023-10-05T21:11:58.988 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.985+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=CA709F 2023-10-05T21:11:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.987+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.987+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.987+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.987+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.987+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.987+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:20DAD6 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.987+0000 7fbf95347680 10 manifest: total_size = 643528 2023-10-05T21:11:58.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.987+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.6043279493989645968 2023-10-05T21:11:58.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.987+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:20DAD6 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.987+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.987+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.9 2023-10-05T21:11:58.990 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.988+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=DAA22E 2023-10-05T21:11:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.990+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.990+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.990+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.990+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.990+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.990+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:666619 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.990+0000 7fbf95347680 10 manifest: total_size = 643064 2023-10-05T21:11:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.990+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.12091857361227265534 2023-10-05T21:11:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.990+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:666619 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.990+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.993 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.990+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.9 2023-10-05T21:11:58.993 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.990+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=20DAD6 2023-10-05T21:11:58.994 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.992+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.995 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.992+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.995 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.992+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.995 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.992+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.995 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.992+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.995 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.992+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:DA6EFA state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.995 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.993+0000 7fbf95347680 10 manifest: total_size = 399275 2023-10-05T21:11:58.995 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.993+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.11393125244019372849 2023-10-05T21:11:58.995 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.993+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:DA6EFA state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.995 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.993+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.995 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.993+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.9 2023-10-05T21:11:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.993+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=666619 2023-10-05T21:11:58.997 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.995+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.997 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.995+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:58.997 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.995+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:58.997 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.995+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:58.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.995+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:58.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.995+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:01BBD6 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.995+0000 7fbf95347680 10 manifest: total_size = 575322 2023-10-05T21:11:58.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.995+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.4038666587012959659 2023-10-05T21:11:58.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.995+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:01BBD6 state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:58.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.995+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:58.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.995+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.10 2023-10-05T21:11:58.998 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.996+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=DA6EFA 2023-10-05T21:11:58.999 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.997+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:58.999 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.997+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:59.000 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.997+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:59.000 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.997+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:59.000 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.997+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:59.000 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.997+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:062A5C state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:59.000 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.998+0000 7fbf95347680 10 manifest: total_size = 117 2023-10-05T21:11:59.000 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.998+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.11423814915295023761 2023-10-05T21:11:59.000 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.998+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:062A5C state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:59.000 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.998+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:59.000 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.998+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.10 2023-10-05T21:11:59.001 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:58.998+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=01BBD6 2023-10-05T21:11:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.000+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.000+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.000+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.000+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.000+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:59.003 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.000+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:2D52AA state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:59.003 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.000+0000 7fbf95347680 10 manifest: total_size = 359837 2023-10-05T21:11:59.003 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.000+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.2678862848998708311 2023-10-05T21:11:59.003 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.000+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:2D52AA state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:59.003 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.000+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:59.003 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.000+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.10 2023-10-05T21:11:59.003 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.001+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=062A5C 2023-10-05T21:11:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.003+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.003+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.003+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.003+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.003+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.003+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:931CFA state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.003+0000 7fbf95347680 10 manifest: total_size = 180000 2023-10-05T21:11:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.003+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.16515934001270856685 2023-10-05T21:11:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.003+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:931CFA state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.003+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.003+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.10 2023-10-05T21:11:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.004+0000 7fbf8641b700 20 handle_completion(): completion ok for obj=2D52AA 2023-10-05T21:11:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.005+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.005+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.005+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.005+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.005+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.005+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:DBFBFB state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.005+0000 7fbf95347680 10 manifest: total_size = 399054 2023-10-05T21:11:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.005+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.13405114963230082370 2023-10-05T21:11:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.005+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=test:DBFBFB state=0x56502f4ba1f8 s->prefetch_data=0 2023-10-05T21:11:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.005+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f4ba1f8 2023-10-05T21:11:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.005+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538.10 2023-10-05T21:11:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.006+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=931CFA 2023-10-05T21:11:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.008+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T21:11:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.008+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T21:11:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.008+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.008+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.008+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.008+0000 7fbf95347680 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T21:11:59.011 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.008+0000 7fbf95347680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :test[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T21:11:59.011 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.008+0000 7fbf95347680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T21:11:59.011 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.009+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=DBFBFB 2023-10-05T21:11:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.009+0000 7fbf95347680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T21:11:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.009+0000 7fbf95347680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T21:11:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.009+0000 7fbf95347680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T21:11:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.009+0000 7fbf95347680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T21:11:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.009+0000 7fbf95347680 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538])"; uploads.size()=0; is_truncated=0 2023-10-05T21:11:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.009+0000 7fbf95347680 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:test[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538]) 2023-10-05T21:11:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.009+0000 7fbf95347680 20 lifecycle: get_system_obj_state: rctx=0x7fff67b07548 obj=default.rgw.meta:root:test state=0x56502f222240 s->prefetch_data=0 2023-10-05T21:11:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.009+0000 7fbf95347680 10 lifecycle: cache get: name=default.rgw.meta+root+test : miss 2023-10-05T21:11:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.010+0000 7fbf95347680 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-10-05T21:11:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.010+0000 7fbf95347680 10 lifecycle: adding default.rgw.meta+root+test to cache LRU end 2023-10-05T21:11:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.010+0000 7fbf95347680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.010+0000 7fbf95347680 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T21:11:59.013 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.010+0000 7fbf95347680 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T21:11:59.013 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.010+0000 7fbf95347680 10 lifecycle: cache get: name=default.rgw.meta+root+test : type miss (requested=0x13, cached=0x16) 2023-10-05T21:11:59.013 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.010+0000 7fbf95347680 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T21:11:59.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.010+0000 7fbf95347680 20 lifecycle: rados_obj.operate() r=0 bl.length=186 2023-10-05T21:11:59.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.010+0000 7fbf95347680 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x13 2023-10-05T21:11:59.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.010+0000 7fbf95347680 10 lifecycle: moving default.rgw.meta+root+test to cache LRU end 2023-10-05T21:11:59.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.010+0000 7fbf95347680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:59.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.011+0000 7fbf95347680 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 info.flags=0x17 2023-10-05T21:11:59.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.011+0000 7fbf95347680 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 to cache LRU end 2023-10-05T21:11:59.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.011+0000 7fbf95347680 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T21:11:59.014 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.011+0000 7fbf95347680 10 lifecycle: distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538, ofs0, ns] 2023-10-05T21:11:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.011+0000 7fbf85c1a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823387 cookie 94902389751152 notifier 4246 bl.length()=800 2023-10-05T21:11:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.011+0000 7fbf85c1a700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 info.flags=0x17 2023-10-05T21:11:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.011+0000 7fbf85c1a700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 to cache LRU end 2023-10-05T21:11:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.011+0000 7fbf85c1a700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T21:11:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.012+0000 7fbf95347680 20 lifecycle: handle_bi_update(): bucket=:test[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538]): orig_sources= new_sources= 2023-10-05T21:11:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.012+0000 7fbf95347680 20 lifecycle: handle_bi_update(): bucket=:test[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538]): potential sources added=[] removed=[] 2023-10-05T21:11:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.012+0000 7fbf95347680 20 lifecycle: handle_bi_update(): bucket=:test[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538]): orig_dests= new_dests= 2023-10-05T21:11:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.012+0000 7fbf95347680 20 lifecycle: handle_bi_update(): bucket=:test[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538]): potential dests added=[] removed=[] 2023-10-05T21:11:59.019 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.017+0000 7fbf95347680 20 get_system_obj_state: rctx=0x7fff67b06fa8 obj=default.rgw.meta:root:test state=0x56502f222240 s->prefetch_data=0 2023-10-05T21:11:59.019 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.017+0000 7fbf95347680 10 cache get: name=default.rgw.meta+root+test : type miss (requested=0x16, cached=0x13) 2023-10-05T21:11:59.019 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.017+0000 7fbf95347680 10 cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-10-05T21:11:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.017+0000 7fbf95347680 10 moving default.rgw.meta+root+test to cache LRU end 2023-10-05T21:11:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.017+0000 7fbf95347680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.017+0000 7fbf95347680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T21:11:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.017+0000 7fbf95347680 10 cache get: name=default.rgw.meta+root+test : hit (requested=0x11, cached=0x17) 2023-10-05T21:11:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.017+0000 7fbf95347680 10 removing default.rgw.meta+root+test from cache 2023-10-05T21:11:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.017+0000 7fbf95347680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:test, ofs0, ns] 2023-10-05T21:11:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.017+0000 7fbf8641b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823388 cookie 94902389764624 notifier 4246 bl.length()=148 2023-10-05T21:11:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.019+0000 7fbf95347680 10 removing default.rgw.meta+root+.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 from cache 2023-10-05T21:11:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.019+0000 7fbf95347680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538, ofs0, ns] 2023-10-05T21:11:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.019+0000 7fbf8641b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823389 cookie 94902389751152 notifier 4246 bl.length()=207 2023-10-05T21:11:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.027+0000 7fbf95347680 10 removing default.rgw.log++pubsub..bucket.test/bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538 from cache 2023-10-05T21:11:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.027+0000 7fbf95347680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test/bb6df8cb-740d-4b21-924c-79673a5d256e.4206.538, ofs0, ns] 2023-10-05T21:11:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.027+0000 7fbf8641b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823646 cookie 94902389740592 notifier 4246 bl.length()=204 2023-10-05T21:11:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.029+0000 7fbf95347680 20 get_system_obj_state: rctx=0x7fff67b07a70 obj=default.rgw.meta:root:.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 state=0x56502f222240 s->prefetch_data=0 2023-10-05T21:11:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.029+0000 7fbf95347680 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 : miss 2023-10-05T21:11:59.032 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 info.flags=0x16 2023-10-05T21:11:59.032 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 10 adding default.rgw.meta+root+.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 to cache LRU end 2023-10-05T21:11:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T21:11:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T21:11:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 20 Read xattr: user.rgw.acl 2023-10-05T21:11:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 20 Read xattr: user.rgw.idtag 2023-10-05T21:11:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 : type miss (requested=0x13, cached=0x16) 2023-10-05T21:11:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 20 rados_obj.operate() r=0 bl.length=369 2023-10-05T21:11:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 info.flags=0x13 2023-10-05T21:11:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 10 moving default.rgw.meta+root+.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 to cache LRU end 2023-10-05T21:11:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T21:11:59.034 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 2023-10-05T21:11:59.034 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.030+0000 7fbf95347680 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :testbuck2[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T21:11:59.035 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.033+0000 7fbf95347680 10 cls_bucket_list_unordered: got 91EA7A 2023-10-05T21:11:59.035 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.034+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=testbuck2:91EA7A state=0x56502f465df8 s->prefetch_data=0 2023-10-05T21:11:59.036 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.034+0000 7fbf95347680 10 manifest: total_size = 399052 2023-10-05T21:11:59.036 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.034+0000 7fbf95347680 20 get_obj_state: setting s->obj_tag to bb6df8cb-740d-4b21-924c-79673a5d256e.4204.10450209940655327179 2023-10-05T21:11:59.036 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.034+0000 7fbf95347680 20 get_obj_state: rctx=0x7fff67b07a80 obj=testbuck2:91EA7A state=0x56502f465df8 s->prefetch_data=0 2023-10-05T21:11:59.036 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.034+0000 7fbf95347680 20 prepare_atomic_modification: state is not atomic. state=0x56502f465df8 2023-10-05T21:11:59.036 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.034+0000 7fbf95347680 20 bucket index object: default.rgw.buckets.index:.dir.bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539.9 2023-10-05T21:11:59.038 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.testbuck2 : miss 2023-10-05T21:11:59.038 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.039 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.039 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 10 cache put: name=default.rgw.log++bucket.sync-source-hints.testbuck2 info.flags=0x0 2023-10-05T21:11:59.039 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 10 adding default.rgw.log++bucket.sync-source-hints.testbuck2 to cache LRU end 2023-10-05T21:11:59.039 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.testbuck2 : miss 2023-10-05T21:11:59.039 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.039 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.039 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 10 cache put: name=default.rgw.log++bucket.sync-target-hints.testbuck2 info.flags=0x0 2023-10-05T21:11:59.039 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 10 adding default.rgw.log++bucket.sync-target-hints.testbuck2 to cache LRU end 2023-10-05T21:11:59.039 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 10 chain_cache_entry: cache_locator= 2023-10-05T21:11:59.039 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T21:11:59.039 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T21:11:59.040 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T21:11:59.040 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :testbuck2[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T21:11:59.040 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.036+0000 7fbf95347680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T21:11:59.041 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf85c1a700 20 handle_completion(): completion ok for obj=91EA7A 2023-10-05T21:11:59.041 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T21:11:59.041 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T21:11:59.041 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T21:11:59.041 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T21:11:59.041 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":testbuck2[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539])"; uploads.size()=0; is_truncated=0 2023-10-05T21:11:59.041 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:testbuck2[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539]) 2023-10-05T21:11:59.041 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 20 lifecycle: get_system_obj_state: rctx=0x7fff67b07548 obj=default.rgw.meta:root:testbuck2 state=0x56502f222240 s->prefetch_data=0 2023-10-05T21:11:59.041 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : miss 2023-10-05T21:11:59.041 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-10-05T21:11:59.042 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 10 lifecycle: adding default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T21:11:59.042 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:59.042 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T21:11:59.042 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T21:11:59.042 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x13, cached=0x16) 2023-10-05T21:11:59.042 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T21:11:59.042 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 20 lifecycle: rados_obj.operate() r=0 bl.length=191 2023-10-05T21:11:59.042 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x13 2023-10-05T21:11:59.042 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 10 lifecycle: moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T21:11:59.042 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.038+0000 7fbf95347680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:59.043 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.040+0000 7fbf95347680 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 info.flags=0x17 2023-10-05T21:11:59.043 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.040+0000 7fbf95347680 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 to cache LRU end 2023-10-05T21:11:59.043 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.040+0000 7fbf95347680 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T21:11:59.043 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.040+0000 7fbf95347680 10 lifecycle: distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539, ofs0, ns] 2023-10-05T21:11:59.044 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.040+0000 7fbf85c1a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823390 cookie 94902389751152 notifier 4246 bl.length()=810 2023-10-05T21:11:59.044 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.040+0000 7fbf85c1a700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 info.flags=0x17 2023-10-05T21:11:59.044 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.040+0000 7fbf85c1a700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 to cache LRU end 2023-10-05T21:11:59.044 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.040+0000 7fbf85c1a700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T21:11:59.044 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.040+0000 7fbf95347680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539]): orig_sources= new_sources= 2023-10-05T21:11:59.044 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.040+0000 7fbf95347680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539]): potential sources added=[] removed=[] 2023-10-05T21:11:59.044 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.040+0000 7fbf95347680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539]): orig_dests= new_dests= 2023-10-05T21:11:59.044 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.040+0000 7fbf95347680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539]): potential dests added=[] removed=[] 2023-10-05T21:11:59.047 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.045+0000 7fbf95347680 20 get_system_obj_state: rctx=0x7fff67b06fa8 obj=default.rgw.meta:root:testbuck2 state=0x56502f222240 s->prefetch_data=0 2023-10-05T21:11:59.047 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.045+0000 7fbf95347680 10 cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x16, cached=0x13) 2023-10-05T21:11:59.047 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.045+0000 7fbf95347680 10 cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-10-05T21:11:59.048 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.045+0000 7fbf95347680 10 moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T21:11:59.048 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.045+0000 7fbf95347680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:59.048 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.045+0000 7fbf95347680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T21:11:59.048 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.045+0000 7fbf95347680 10 cache get: name=default.rgw.meta+root+testbuck2 : hit (requested=0x11, cached=0x17) 2023-10-05T21:11:59.048 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.045+0000 7fbf95347680 10 removing default.rgw.meta+root+testbuck2 from cache 2023-10-05T21:11:59.048 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.045+0000 7fbf95347680 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:testbuck2, ofs0, ns] 2023-10-05T21:11:59.048 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.046+0000 7fbf8641b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823391 cookie 94902389763456 notifier 4246 bl.length()=153 2023-10-05T21:11:59.049 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.047+0000 7fbf95347680 10 removing default.rgw.meta+root+.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 from cache 2023-10-05T21:11:59.049 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.047+0000 7fbf95347680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.testbuck2:bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539, ofs0, ns] 2023-10-05T21:11:59.049 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.047+0000 7fbf85c1a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823392 cookie 94902389751152 notifier 4246 bl.length()=212 2023-10-05T21:11:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.055+0000 7fbf95347680 10 removing default.rgw.log++pubsub..bucket.testbuck2/bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539 from cache 2023-10-05T21:11:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.055+0000 7fbf95347680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.testbuck2/bb6df8cb-740d-4b21-924c-79673a5d256e.4206.539, ofs0, ns] 2023-10-05T21:11:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.055+0000 7fbf85c1a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823647 cookie 94902389740592 notifier 4246 bl.length()=209 2023-10-05T21:11:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.058+0000 7fbf95347680 10 removing key index: DNEBPVGXKFWAQJAQIZSN 2023-10-05T21:11:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.058+0000 7fbf95347680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.keys:DNEBPVGXKFWAQJAQIZSN, ofs0, ns] 2023-10-05T21:11:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.058+0000 7fbf8641b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823648 cookie 94902389767760 notifier 4246 bl.length()=170 2023-10-05T21:11:59.061 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.060+0000 7fbf95347680 10 removing email index: foo.client.0+test@test.test 2023-10-05T21:11:59.062 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.060+0000 7fbf95347680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-10-05T21:11:59.062 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.060+0000 7fbf85c1a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823393 cookie 94902389764624 notifier 4246 bl.length()=178 2023-10-05T21:11:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.061+0000 7fbf95347680 10 removing user buckets index 2023-10-05T21:11:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.061+0000 7fbf95347680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2023-10-05T21:11:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.061+0000 7fbf8641b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823394 cookie 94902389751152 notifier 4246 bl.length()=169 2023-10-05T21:11:59.065 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.063+0000 7fbf95347680 10 removing user index: foo.client.0 2023-10-05T21:11:59.065 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.063+0000 7fbf95347680 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2023-10-05T21:11:59.065 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.063+0000 7fbf95347680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T21:11:59.065 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.063+0000 7fbf85c1a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823649 cookie 94902389740592 notifier 4246 bl.length()=161 2023-10-05T21:11:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.066+0000 7fbf95347680 20 remove_watcher() i=0 2023-10-05T21:11:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.066+0000 7fbf95347680 2 removed watcher, disabling cache 2023-10-05T21:11:59.068 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.066+0000 7fbf95347680 20 remove_watcher() i=1 2023-10-05T21:11:59.069 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.067+0000 7fbf95347680 20 remove_watcher() i=2 2023-10-05T21:11:59.070 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.068+0000 7fbf95347680 20 remove_watcher() i=3 2023-10-05T21:11:59.070 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.069+0000 7fbf95347680 20 remove_watcher() i=4 2023-10-05T21:11:59.071 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.069+0000 7fbf95347680 20 remove_watcher() i=5 2023-10-05T21:11:59.072 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.070+0000 7fbf95347680 20 remove_watcher() i=6 2023-10-05T21:11:59.073 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.071+0000 7fbf95347680 20 remove_watcher() i=7 2023-10-05T21:11:59.073 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.072+0000 7fbf59fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:11:59.079 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2023-10-05T21:11:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setuser ceph since I am not root 2023-10-05T21:11:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T21:11:59.167 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.165+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.166+0000 7f6bc824e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.168 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.166+0000 7f6bc824e680 20 realm 2023-10-05T21:11:59.169 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.166+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.169 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.166+0000 7f6bc824e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.169 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.166+0000 7f6bc824e680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T21:11:59.169 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.166+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.169 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.166+0000 7f6bc824e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.169 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.166+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.169 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.167+0000 7f6bc824e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:59.169 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.167+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.170 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.167+0000 7f6bc824e680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:11:59.170 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.167+0000 7f6bc824e680 20 searching for the correct realm 2023-10-05T21:11:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:11:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:11:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:11:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:11:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:11:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.175+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.176+0000 7f6bc824e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:11:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:11:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:11:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:11:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.184 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:11:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:11:59.185 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.182+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.186 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.186 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 20 zone default found 2023-10-05T21:11:59.187 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 4 Realm: () 2023-10-05T21:11:59.187 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 4 ZoneGroup: default (cb9d3d40-68a5-4791-bf47-2aada3a09714) 2023-10-05T21:11:59.187 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 4 Zone: default (bb6df8cb-740d-4b21-924c-79673a5d256e) 2023-10-05T21:11:59.187 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T21:11:59.187 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 20 zonegroup default 2023-10-05T21:11:59.187 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.187 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.187 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.187 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.187 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.187 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.188 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 20 started sync module instance, tier type = 2023-10-05T21:11:59.188 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.183+0000 7f6bc824e680 20 started zone id=bb6df8cb-740d-4b21-924c-79673a5d256e (name=default) with tier type = 2023-10-05T21:11:59.192 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.190+0000 7f6bc824e680 20 add_watcher() i=0 2023-10-05T21:11:59.192 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.190+0000 7f6bc824e680 20 add_watcher() i=1 2023-10-05T21:11:59.193 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.190+0000 7f6bc824e680 20 add_watcher() i=2 2023-10-05T21:11:59.193 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.190+0000 7f6bc824e680 20 add_watcher() i=3 2023-10-05T21:11:59.193 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.190+0000 7f6bc824e680 20 add_watcher() i=4 2023-10-05T21:11:59.193 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.190+0000 7f6bc824e680 20 add_watcher() i=5 2023-10-05T21:11:59.193 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.190+0000 7f6bc824e680 20 add_watcher() i=6 2023-10-05T21:11:59.193 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.191+0000 7f6bc824e680 20 add_watcher() i=7 2023-10-05T21:11:59.193 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.191+0000 7f6bc824e680 2 all 8 watchers are set, enabling cache 2023-10-05T21:11:59.194 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.192+0000 7f6b07fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:11:59.194 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.193+0000 7f6bc824e680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T21:11:59.195 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.193+0000 7f6bc824e680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T21:11:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6af6ffd700 20 reqs_thread_entry: start 2023-10-05T21:11:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6af4ff9700 20 reqs_thread_entry: start 2023-10-05T21:11:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6bc824e680 20 init_complete bucket index max shards: 11 2023-10-05T21:11:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6bc824e680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T21:11:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6aeeffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 21:12:29 2023 2023-10-05T21:11:59.233 INFO:teuthology.orchestra.run.smithi070.stderr: 2023-10-05T21:11:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6aee7fc700 20 reqs_thread_entry: start 2023-10-05T21:11:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6bc824e680 20 get_system_obj_state: rctx=0x7ffe1609f768 obj=default.rgw.meta:users.uid:bar.client.0 state=0x5573130ab240 s->prefetch_data=0 2023-10-05T21:11:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6bc824e680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T21:11:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6bc824e680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-10-05T21:11:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6bc824e680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T21:11:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6bc824e680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6bc824e680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T21:11:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6bc824e680 20 Read xattr: user.rgw.idtag 2023-10-05T21:11:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6bc824e680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T21:11:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.230+0000 7f6bc824e680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.231+0000 7f6bc824e680 20 rados_obj.operate() r=0 bl.length=469 2023-10-05T21:11:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.231+0000 7f6bc824e680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-10-05T21:11:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.231+0000 7f6bc824e680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T21:11:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.231+0000 7f6bc824e680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.231+0000 7f6bc824e680 10 removing key index: XWTSCMQDUCFJOSOPVZAH 2023-10-05T21:11:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.231+0000 7f6bc824e680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:XWTSCMQDUCFJOSOPVZAH, ofs0, ns] 2023-10-05T21:11:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.231+0000 7f6bb9322700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823395 cookie 93952730673456 notifier 4252 bl.length()=170 2023-10-05T21:11:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.232+0000 7f6bc824e680 10 removing email index: bar.client.0+test@test.test 2023-10-05T21:11:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.232+0000 7f6bc824e680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-10-05T21:11:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.233+0000 7f6bb8b21700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823396 cookie 93952730673456 notifier 4252 bl.length()=178 2023-10-05T21:11:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.234+0000 7f6bc824e680 10 removing user buckets index 2023-10-05T21:11:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.234+0000 7f6bc824e680 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2023-10-05T21:11:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.234+0000 7f6bb9322700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824383 cookie 93952730679568 notifier 4252 bl.length()=169 2023-10-05T21:11:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.235+0000 7f6bc824e680 10 removing user index: bar.client.0 2023-10-05T21:11:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.235+0000 7f6bc824e680 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2023-10-05T21:11:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.235+0000 7f6bc824e680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T21:11:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.236+0000 7f6bb8b21700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823397 cookie 93952730673456 notifier 4252 bl.length()=161 2023-10-05T21:11:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.238+0000 7f6bc824e680 20 remove_watcher() i=0 2023-10-05T21:11:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.238+0000 7f6bc824e680 2 removed watcher, disabling cache 2023-10-05T21:11:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.239+0000 7f6bc824e680 20 remove_watcher() i=1 2023-10-05T21:11:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.239+0000 7f6bc824e680 20 remove_watcher() i=2 2023-10-05T21:11:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.240+0000 7f6bc824e680 20 remove_watcher() i=3 2023-10-05T21:11:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.241+0000 7f6bc824e680 20 remove_watcher() i=4 2023-10-05T21:11:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.242+0000 7f6bc824e680 20 remove_watcher() i=5 2023-10-05T21:11:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.243+0000 7f6bc824e680 20 remove_watcher() i=6 2023-10-05T21:11:59.245 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.243+0000 7f6bc824e680 20 remove_watcher() i=7 2023-10-05T21:11:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.244+0000 7f6b07fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:11:59.251 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2023-10-05T21:11:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setuser ceph since I am not root 2023-10-05T21:11:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T21:11:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.340+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.341+0000 7fba45c58680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.341+0000 7fba45c58680 20 realm 2023-10-05T21:11:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.341+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.341+0000 7fba45c58680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.341+0000 7fba45c58680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T21:11:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.341+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.341+0000 7fba45c58680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.341+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.341+0000 7fba45c58680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.341+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.342+0000 7fba45c58680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:11:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.342+0000 7fba45c58680 20 searching for the correct realm 2023-10-05T21:11:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.349+0000 7fba45c58680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:11:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.349+0000 7fba45c58680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:11:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.349+0000 7fba45c58680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:11:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.349+0000 7fba45c58680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:11:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.349+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.349+0000 7fba45c58680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.349+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.350+0000 7fba45c58680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.350+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.350+0000 7fba45c58680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:11:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.350+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.350+0000 7fba45c58680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.350+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.350+0000 7fba45c58680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.350+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.351+0000 7fba45c58680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.356+0000 7fba45c58680 20 RGWRados::pool_iterate: got zonegroup_info.cb9d3d40-68a5-4791-bf47-2aada3a09714 2023-10-05T21:11:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.356+0000 7fba45c58680 20 RGWRados::pool_iterate: got zone_info.bb6df8cb-740d-4b21-924c-79673a5d256e 2023-10-05T21:11:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.356+0000 7fba45c58680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T21:11:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.356+0000 7fba45c58680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T21:11:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.356+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.357+0000 7fba45c58680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.357+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.357+0000 7fba45c58680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T21:11:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.357+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.357+0000 7fba45c58680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T21:11:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.357+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.357+0000 7fba45c58680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T21:11:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.357+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 20 zone default found 2023-10-05T21:11:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 4 Realm: () 2023-10-05T21:11:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 4 ZoneGroup: default (cb9d3d40-68a5-4791-bf47-2aada3a09714) 2023-10-05T21:11:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 4 Zone: default (bb6df8cb-740d-4b21-924c-79673a5d256e) 2023-10-05T21:11:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T21:11:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 20 zonegroup default 2023-10-05T21:11:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T21:11:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 20 started sync module instance, tier type = 2023-10-05T21:11:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.358+0000 7fba45c58680 20 started zone id=bb6df8cb-740d-4b21-924c-79673a5d256e (name=default) with tier type = 2023-10-05T21:11:59.368 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.366+0000 7fba45c58680 20 add_watcher() i=0 2023-10-05T21:11:59.368 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.366+0000 7fba45c58680 20 add_watcher() i=1 2023-10-05T21:11:59.368 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.366+0000 7fba45c58680 20 add_watcher() i=2 2023-10-05T21:11:59.368 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.366+0000 7fba45c58680 20 add_watcher() i=3 2023-10-05T21:11:59.368 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.366+0000 7fba45c58680 20 add_watcher() i=4 2023-10-05T21:11:59.368 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.366+0000 7fba45c58680 20 add_watcher() i=5 2023-10-05T21:11:59.368 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.366+0000 7fba45c58680 20 add_watcher() i=6 2023-10-05T21:11:59.369 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.366+0000 7fba45c58680 20 add_watcher() i=7 2023-10-05T21:11:59.369 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.366+0000 7fba45c58680 2 all 8 watchers are set, enabling cache 2023-10-05T21:11:59.370 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.368+0000 7fba09fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:11:59.370 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.368+0000 7fba45c58680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T21:11:59.370 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.368+0000 7fba45c58680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T21:11:59.408 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fb980ff9700 20 reqs_thread_entry: start 2023-10-05T21:11:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fb9767fc700 20 reqs_thread_entry: start 2023-10-05T21:11:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fba45c58680 20 init_complete bucket index max shards: 11 2023-10-05T21:11:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fba45c58680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T21:11:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fb974ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 21:12:29 2023 2023-10-05T21:11:59.409 INFO:teuthology.orchestra.run.smithi070.stderr: 2023-10-05T21:11:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fb95bfff700 20 reqs_thread_entry: start 2023-10-05T21:11:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fba45c58680 20 get_system_obj_state: rctx=0x7ffd6f8a1088 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x5609fad19240 s->prefetch_data=0 2023-10-05T21:11:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fba45c58680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T21:11:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fba45c58680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-10-05T21:11:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fba45c58680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T21:11:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fba45c58680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fba45c58680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T21:11:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fba45c58680 20 Read xattr: user.rgw.idtag 2023-10-05T21:11:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fba45c58680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T21:11:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.406+0000 7fba45c58680 20 rados->read ofs=0 len=0 2023-10-05T21:11:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.407+0000 7fba45c58680 20 rados_obj.operate() r=0 bl.length=528 2023-10-05T21:11:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.407+0000 7fba45c58680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-10-05T21:11:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.407+0000 7fba45c58680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T21:11:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.407+0000 7fba45c58680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T21:11:59.411 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.407+0000 7fba45c58680 10 removing key index: MSXBHLLEJJIOCMIAIOBV 2023-10-05T21:11:59.411 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.407+0000 7fba45c58680 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.keys:MSXBHLLEJJIOCMIAIOBV, ofs0, ns] 2023-10-05T21:11:59.411 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.407+0000 7fba327fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823398 cookie 94600864365664 notifier 4258 bl.length()=170 2023-10-05T21:11:59.411 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.408+0000 7fba45c58680 10 removing email index: testx$tenanteduser.client.0+test@test.test 2023-10-05T21:11:59.411 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.408+0000 7fba45c58680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-10-05T21:11:59.412 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.409+0000 7fba31ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823650 cookie 94600864369328 notifier 4258 bl.length()=193 2023-10-05T21:11:59.412 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.410+0000 7fba45c58680 10 removing user buckets index 2023-10-05T21:11:59.412 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.410+0000 7fba45c58680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2023-10-05T21:11:59.412 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.410+0000 7fba31ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823399 cookie 94600864373072 notifier 4258 bl.length()=184 2023-10-05T21:11:59.413 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.411+0000 7fba45c58680 10 removing user index: testx$tenanteduser.client.0 2023-10-05T21:11:59.413 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.411+0000 7fba45c58680 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2023-10-05T21:11:59.413 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.411+0000 7fba45c58680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T21:11:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.412+0000 7fba327fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823651 cookie 94600864356640 notifier 4258 bl.length()=176 2023-10-05T21:11:59.416 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.414+0000 7fba45c58680 20 remove_watcher() i=0 2023-10-05T21:11:59.416 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.414+0000 7fba45c58680 2 removed watcher, disabling cache 2023-10-05T21:11:59.416 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.415+0000 7fba45c58680 20 remove_watcher() i=1 2023-10-05T21:11:59.417 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.415+0000 7fba45c58680 20 remove_watcher() i=2 2023-10-05T21:11:59.418 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.416+0000 7fba45c58680 20 remove_watcher() i=3 2023-10-05T21:11:59.419 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.417+0000 7fba45c58680 20 remove_watcher() i=4 2023-10-05T21:11:59.419 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.418+0000 7fba45c58680 20 remove_watcher() i=5 2023-10-05T21:11:59.420 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.419+0000 7fba45c58680 20 remove_watcher() i=6 2023-10-05T21:11:59.421 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.419+0000 7fba45c58680 20 remove_watcher() i=7 2023-10-05T21:11:59.422 INFO:teuthology.orchestra.run.smithi070.stderr:2023-10-05T21:11:59.420+0000 7fba09fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T21:11:59.427 INFO:tasks.s3tests:Removing s3-tests... 2023-10-05T21:11:59.428 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T21:11:59.549 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2023-10-05T21:11:59.564 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-10-05T21:11:59.580 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-10-05T21:11:59.580 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T21:11:59.581 INFO:tasks.rgw.client.0.smithi070.stdout:2023-10-05T21:11:59.579+0000 7f73e5808700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi070.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --rgw_crypt_sse_s3_vault_addr smithi070.front.sepia.ceph.com:8200 --rgw_crypt_sse_s3_vault_token_file /etc/ceph/vault-root-token --foreground (PID: 119023) UID: 0 2023-10-05T21:11:59.581 INFO:tasks.rgw.client.0.smithi070.stdout:2023-10-05T21:11:59.579+0000 7f73f5b97740 -1 shutting down 2023-10-05T21:12:01.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2023-10-05T21:12:05.681 INFO:tasks.rgw.client.0:Stopped 2023-10-05T21:12:05.682 DEBUG:teuthology.orchestra.run.smithi070:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-10-05T21:12:05.708 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/vault-root-token 2023-10-05T21:12:05.783 DEBUG:teuthology.run_tasks:Unwinding manager vault 2023-10-05T21:12:05.792 INFO:tasks.vault:Stopping Vault instance 2023-10-05T21:12:05.793 DEBUG:tasks.vault.client.0:waiting for process to exit 2023-10-05T21:12:05.793 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T21:12:05.825 INFO:tasks.vault.client.0.smithi070.stdout:==> Vault shutdown triggered 2023-10-05T21:12:05.826 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:12:05.825Z [INFO] core: marked as sealed 2023-10-05T21:12:05.826 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:12:05.825Z [INFO] core: pre-seal teardown starting 2023-10-05T21:12:05.826 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:12:05.825Z [INFO] rollback: stopping rollback manager 2023-10-05T21:12:05.826 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:12:05.825Z [INFO] core: pre-seal teardown complete 2023-10-05T21:12:05.826 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:12:05.825Z [INFO] core: stopping cluster listeners 2023-10-05T21:12:05.827 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:12:05.825Z [INFO] core.cluster-listener: forwarding rpc listeners stopped 2023-10-05T21:12:05.934 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:12:05.933Z [INFO] core.cluster-listener: rpc listeners successfully shut down 2023-10-05T21:12:05.934 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:12:05.933Z [INFO] core: cluster listeners successfully shut down 2023-10-05T21:12:05.934 INFO:tasks.vault.client.0.smithi070.stderr:2023-10-05T21:12:05.933Z [INFO] core: vault is sealed 2023-10-05T21:12:05.935 INFO:tasks.vault.client.0:Stopped 2023-10-05T21:12:05.935 INFO:tasks.vault:Removing Vault... 2023-10-05T21:12:05.936 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/vault /home/ubuntu/cephtest/vault.zip 2023-10-05T21:12:06.012 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-05T21:12:06.022 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-10-05T21:12:06.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T21:12:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-10-05T21:12:06.365 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":36,"fsid":"18ea46c7-744d-428a-a77f-8692a242600e","created":"2023-10-05T21:06:59.300586+0000","modified":"2023-10-05T21:09:06.404646+0000","last_up_change":"2023-10-05T21:07:06.668282+0000","last_in_change":"2023-10-05T21:07:00.278515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T21:07:07.011067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T21:07:07.999964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-10-05T21:07:25.666305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-10-05T21:07:27.662111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-10-05T21:07:29.780180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-10-05T21:07:31.374958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-10-05T21:07:33.357071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-10-05T21:07:35.409461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2023-10-05T21:07:46.029389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2023-10-05T21:08:25.620438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"7481092a-4057-4980-8713-86e63cc546cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":114355},{"type":"v1","addr":"172.21.15.70:6801","nonce":114355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":114355},{"type":"v1","addr":"172.21.15.70:6803","nonce":114355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":114355},{"type":"v1","addr":"172.21.15.70:6807","nonce":114355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":114355},{"type":"v1","addr":"172.21.15.70:6805","nonce":114355}]},"public_addr":"172.21.15.70:6801/114355","cluster_addr":"172.21.15.70:6803/114355","heartbeat_back_addr":"172.21.15.70:6807/114355","heartbeat_front_addr":"172.21.15.70:6805/114355","state":["exists","up"]},{"osd":1,"uuid":"cc290433-0501-4fe0-85c0-a82ae80e3cea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":114354},{"type":"v1","addr":"172.21.15.70:6809","nonce":114354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":114354},{"type":"v1","addr":"172.21.15.70:6811","nonce":114354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":114354},{"type":"v1","addr":"172.21.15.70:6815","nonce":114354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":114354},{"type":"v1","addr":"172.21.15.70:6813","nonce":114354}]},"public_addr":"172.21.15.70:6809/114354","cluster_addr":"172.21.15.70:6811/114354","heartbeat_back_addr":"172.21.15.70:6815/114354","heartbeat_front_addr":"172.21.15.70:6813/114354","state":["exists","up"]},{"osd":2,"uuid":"cf0ef326-b20b-48ad-82c7-6ff93ca6770c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":114384},{"type":"v1","addr":"172.21.15.70:6817","nonce":114384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":114384},{"type":"v1","addr":"172.21.15.70:6819","nonce":114384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":114384},{"type":"v1","addr":"172.21.15.70:6823","nonce":114384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":114384},{"type":"v1","addr":"172.21.15.70:6821","nonce":114384}]},"public_addr":"172.21.15.70:6817/114384","cluster_addr":"172.21.15.70:6819/114384","heartbeat_back_addr":"172.21.15.70:6823/114384","heartbeat_front_addr":"172.21.15.70:6821/114384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T21:07:04.586642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T21:07:05.198645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T21:07:04.673759+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.9","mappings":[{"from":0,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"4.15","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":0}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]},{"pgid":"4.27","mappings":[{"from":1,"to":0}]},{"pgid":"4.31","mappings":[{"from":1,"to":2}]},{"pgid":"4.3a","mappings":[{"from":1,"to":0}]},{"pgid":"4.3c","mappings":[{"from":1,"to":2}]},{"pgid":"10.0","mappings":[{"from":2,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T21:12:06.378 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-05T21:12:06.703 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-05T21:12:06.703 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-05T21:12:06.703 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-05T21:12:06.703 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T21:12:06.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2023-10-05T21:12:06.751 INFO:tasks.ceph.osd.0:Stopped 2023-10-05T21:12:06.751 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-05T21:12:06.751 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T21:12:06.796 INFO:tasks.ceph.osd.1:Stopped 2023-10-05T21:12:06.796 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-05T21:12:06.796 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T21:12:06.837 INFO:tasks.ceph.osd.2:Stopped 2023-10-05T21:12:06.837 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-05T21:12:06.837 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-05T21:12:06.837 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T21:12:06.900 INFO:tasks.ceph.mgr.x:Stopped 2023-10-05T21:12:06.900 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-05T21:12:06.900 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-05T21:12:06.900 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T21:12:06.950 INFO:tasks.ceph.mon.a:Stopped 2023-10-05T21:12:06.951 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-05T21:12:06.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T21:12:07.019 INFO:teuthology.orchestra.run.smithi070.stdout:1696540029.842548 mon.a (mon.0) 82 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-05T21:12:07.020 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-05T21:12:07.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T21:12:07.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T21:12:07.157 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T21:12:07.225 INFO:teuthology.orchestra.run.smithi070.stdout:1696540029.842548 mon.a (mon.0) 82 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-05T21:12:07.225 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi070.front.sepia.ceph.com 2023-10-05T21:12:07.225 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-05T21:12:09.048 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi070.front.sepia.ceph.com 2023-10-05T21:12:09.048 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-05T21:12:09.130 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi070.front.sepia.ceph.com 2023-10-05T21:12:09.131 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-05T21:12:09.202 INFO:tasks.ceph:Archiving mon data... 2023-10-05T21:12:09.204 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411774/data/mon.a.tgz 2023-10-05T21:12:09.204 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2023-10-05T21:12:09.222 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.O9HPXMhvlY 2023-10-05T21:12:09.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.O9HPXMhvlY 2023-10-05T21:12:09.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.O9HPXMhvlY 2023-10-05T21:12:09.451 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.O9HPXMhvlY is 285KB 2023-10-05T21:12:09.544 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.O9HPXMhvlY 2023-10-05T21:12:09.562 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-05T21:12:09.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-05T21:12:09.627 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-05T21:12:09.627 DEBUG:teuthology.orchestra.run.smithi070:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-05T21:12:09.695 INFO:teuthology.orchestra.run.smithi070.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-05T21:12:09.701 INFO:tasks.ceph:Archiving crash dumps... 2023-10-05T21:12:09.703 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411774/remote/smithi070/crash 2023-10-05T21:12:09.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-05T21:12:09.767 INFO:tasks.ceph:Compressing logs... 2023-10-05T21:12:09.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T21:12:37.884 INFO:tasks.ceph:Archiving logs... 2023-10-05T21:12:37.885 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411774/remote/smithi070/log 2023-10-05T21:12:37.886 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-05T21:12:41.101 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-05T21:12:41.110 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-05T21:12:41.111 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-05T21:12:41.186 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-05T21:12:41.196 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-05T21:12:41.196 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T21:12:41.220 INFO:teuthology.orchestra.run.smithi070.stderr:bash: ntpq: command not found 2023-10-05T21:12:41.225 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-05T21:12:41.225 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2023-10-05T21:12:41.226 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 83 -705us[ -705us] +/- 84ms 2023-10-05T21:12:41.226 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv02.front.sepia.ceph.com 3 8 377 212 -778us[ -787us] +/- 58ms 2023-10-05T21:12:41.226 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +1292us[+1292us] +/- 66ms 2023-10-05T21:12:41.226 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T21:12:41.227 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-05T21:12:41.236 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-05T21:12:41.236 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-05T21:12:41.245 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-05T21:12:41.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-05T21:12:41.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T21:12:41.397 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-05T21:12:41.406 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-05T21:12:41.545 INFO:teuthology.task.internal:Duration was 4265.433845 seconds 2023-10-05T21:12:41.545 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-05T21:12:41.578 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-05T21:12:41.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-05T21:12:41.625 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-05T21:12:41.852 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-05T21:12:41.852 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2023-10-05T21:12:41.852 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-05T21:12:41.873 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-05T21:12:41.874 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T21:12:41.951 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-05T21:12:41.952 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-05T21:12:42.058 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-05T21:12:42.068 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-05T21:12:42.069 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-05T21:12:42.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-05T21:12:42.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-05T21:12:42.163 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2023-10-05T21:12:42.188 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-05T21:12:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T21:12:42.245 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-05T21:12:42.256 INFO:teuthology.task.internal:Transferring archived files... 2023-10-05T21:12:42.256 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411774/remote/smithi070 2023-10-05T21:12:42.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-05T21:12:42.325 INFO:teuthology.task.internal:Removing archive directory... 2023-10-05T21:12:42.325 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-05T21:12:42.376 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-05T21:12:42.386 INFO:teuthology.task.internal:Not uploading archives. 2023-10-05T21:12:42.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-05T21:12:42.396 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-05T21:12:42.396 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-05T21:12:42.433 INFO:teuthology.orchestra.run.smithi070.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 5 21:12 /home/ubuntu/cephtest 2023-10-05T21:12:42.435 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-05T21:12:42.446 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-05T21:12:42.467 INFO:teuthology.nuke:Checking targets against current locks 2023-10-05T21:12:42.492 DEBUG:teuthology.nuke:shortname: smithi070 2023-10-05T21:12:42.492 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T21:12:42.517 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411774', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:43:47.353685', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8z/vefdvZUw41n3xNDmfIeVLkjdixVzTHNkrvMPaVIX3CjFH+aUP2pxblFIoZ876J04OLGO6n7SqTluvifork='} 2023-10-05T21:12:42.556 INFO:teuthology.orchestra.console.smithi070:Power off 2023-10-05T21:12:42.557 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-05T21:12:42.582 DEBUG:teuthology.orchestra.console.smithi070:power off output: Chassis Power Control: Down/Off 2023-10-05T21:12:42.582 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T21:12:42.606 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2023-10-05T21:12:46.608 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T21:12:46.718 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2023-10-05T21:12:50.722 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T21:12:50.832 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is off 2023-10-05T21:12:50.933 INFO:teuthology.orchestra.console.smithi070:Power off completed 2023-10-05T21:12:51.037 INFO:teuthology.run:Summary data: description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} duration: 4265.433845281601 failure_reason: '"1696540029.842548 mon.a (mon.0) 82 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2023-10-05T21:12:51.037 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-05T21:12:51.115 INFO:teuthology.run:FAIL