2023-01-24T17:15:11.499 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-24T17:15:11.500 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135503 --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135503/orig.config.yaml 2023-01-24T17:15:11.532 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T17:15:11.603 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135503 branch: wip-yuri7-testing-2023-01-23-1532-quincy description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} email: yweinstein@radhat.com first_in_suite: false job_id: '7135503' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri7-testing-2023-01-23-1532-quincy ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: vault rgw crypt vault auth: token rgw crypt vault prefix: /v1/transit/export/encryption-key/ rgw crypt vault secret engine: transit setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 49f8fb05584886826e8eade75f7105fba754560c wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: default sha1: 49f8fb05584886826e8eade75f7105fba754560c rgw: client.0: use-vault-role: client.0 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri7-testing-2023-01-23-1532-quincy sha1: 49f8fb05584886826e8eade75f7105fba754560c owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 49f8fb05584886826e8eade75f7105fba754560c sleep_before_teardown: 0 suite: rgw suite_branch: wip-yuri7-testing-2023-01-23-1532-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 49f8fb05584886826e8eade75f7105fba754560c targets: smithi077.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjvvfvCe9LCqROmpxYy9mNZms0xnbU9bEpnlulym6um/jVtB8UK2T0VosXV51jyFB6VLBov5o2EIMYIOTn95U0iWcC8Qhy4DbR2o4becRkORALtzBdfb4zZLO6lQq1wnn5s9eIFHcMETtf+93jGiObxCkrYrfMYEeGuBiHykJwLLTWgGZmxVAUISBc6GjG/HrjXvrczl9ahaWMVQ8KPxP5x0R+aRo/HHUH9DORY4GVcoCaQsh0DyCKJ/bPY2tFgsqR7XEiwvgTftzKLYboZPRO6mE82QIt2QXb3ynrxMGIdL6ZCSNqxoaxkJz1YVXS4JGHy8TLOnczYyI+iuHqR48NRJZdC/antLW+flG//2mbo56YV2zLE+icC7mn8HtBtQkO9OpNlKTFA8uulAmUftDVGMHmp0NprqCLxeja9pMnnQ+wFLV1mnhAGhjMgN6Q+P5T/BjVBKI0vfjBWWLxu+vXuOX/1V6iSICkNsrg5n6kT5VBuC1gJN4EY9Z8W98Juxs= tasks: - install: null - ceph: null - vault: client.0: engine: transit flavor: old install_sha256: 7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 install_url: https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip prefix: /v1/transit/keys/ root_token: test_root_token secrets: - path: my-key-1 - path: my-key-2 - rgw: client.0: null - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 force-branch: ceph-quincy vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_16:04:03 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-24T17:15:11.603 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa; will attempt to use it 2023-01-24T17:15:11.604 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa/tasks 2023-01-24T17:15:11.604 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-24T17:15:11.606 INFO:teuthology.task.internal:Checking packages... 2023-01-24T17:15:11.633 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '49f8fb05584886826e8eade75f7105fba754560c' 2023-01-24T17:15:11.633 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-24T17:15:11.634 INFO:teuthology.packaging:ref: None 2023-01-24T17:15:11.634 INFO:teuthology.packaging:tag: None 2023-01-24T17:15:11.634 INFO:teuthology.packaging:branch: wip-yuri7-testing-2023-01-23-1532-quincy 2023-01-24T17:15:11.635 INFO:teuthology.packaging:sha1: 49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T17:15:11.635 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri7-testing-2023-01-23-1532-quincy 2023-01-24T17:15:11.784 INFO:teuthology.task.internal:Found packages for ceph version 17.2.5-727.g49f8fb05 2023-01-24T17:15:11.784 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-24T17:15:11.796 INFO:teuthology.task.internal:no buildpackages task found 2023-01-24T17:15:11.796 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-24T17:15:11.808 INFO:teuthology.task.internal:Saving configuration 2023-01-24T17:15:11.820 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-24T17:15:11.827 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-24T17:15:11.855 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135503', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 17:08:39.285855', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPMO6YCB6Q2BmwMiFsQB1flKxfN2ndqzCMO+D479js/q/d8elTD0iQeZoSXplzCRrgvNsBsBR50iuldUfjpFJIs='} 2023-01-24T17:15:11.856 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-24T17:15:11.862 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-01-24T17:15:11.862 INFO:teuthology.run_tasks:Running task console_log... 2023-01-24T17:15:11.910 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb775cb65e0>, signals=[15]) 2023-01-24T17:15:11.911 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-24T17:15:11.921 INFO:teuthology.task.internal:Opening connections... 2023-01-24T17:15:11.921 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2023-01-24T17:15:11.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T17:15:11.992 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-24T17:15:12.000 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2023-01-24T17:15:12.019 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2023-01-24T17:15:12.019 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2023-01-24T17:15:12.076 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2023-01-24T17:15:12.077 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="8" 2023-01-24T17:15:12.077 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2023-01-24T17:15:12.077 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2023-01-24T17:15:12.077 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="8" 2023-01-24T17:15:12.077 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el8" 2023-01-24T17:15:12.078 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-24T17:15:12.078 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2023-01-24T17:15:12.078 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-24T17:15:12.078 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2023-01-24T17:15:12.078 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-24T17:15:12.079 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-24T17:15:12.079 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-24T17:15:12.079 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2023-01-24T17:15:12.105 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-24T17:15:12.113 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-24T17:15:12.118 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-24T17:15:12.119 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2023-01-24T17:15:12.136 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-24T17:15:12.143 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-24T17:15:12.143 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2023-01-24T17:15:12.311 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-24T17:15:12.347 INFO:teuthology.run_tasks:Running task kernel... 2023-01-24T17:15:12.363 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-24T17:15:12.365 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-24T17:15:12.366 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-24T17:15:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T17:15:12.386 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2023-01-24T17:15:12.446 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-408.el8.x86_64 2023-01-24T17:15:12.447 INFO:teuthology.task.kernel:Running kernel on smithi077: 4.18.0-408.el8.x86_64 2023-01-24T17:15:12.447 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2023-01-24T17:15:15.986 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - AppStream 40 MB/s | 27 MB 00:00 2023-01-24T17:15:21.644 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - BaseOS 39 MB/s | 27 MB 00:00 2023-01-24T17:15:26.035 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - Extras 61 kB/s | 18 kB 00:00 2023-01-24T17:15:26.445 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - PowerTools 20 MB/s | 5.6 MB 00:00 2023-01-24T17:15:27.853 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2023-01-24T17:15:29.070 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 12 MB/s | 13 MB 00:01 2023-01-24T17:15:32.437 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 184 kB/s | 24 kB 00:00 2023-01-24T17:15:35.518 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2023-01-24T17:15:35.519 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2023-01-24T17:15:35.679 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:15:35.679 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:15:35.679 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-01-24T17:15:35.680 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:15:35.680 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:15:35.680 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 4.18.0-448.el8 CentOS-BaseOS 9.2 M 2023-01-24T17:15:35.680 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core x86_64 4.18.0-448.el8 CentOS-BaseOS 41 M 2023-01-24T17:15:35.681 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules x86_64 4.18.0-448.el8 CentOS-BaseOS 34 M 2023-01-24T17:15:35.681 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:15:35.681 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:15:35.681 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:15:35.681 INFO:teuthology.orchestra.run.smithi077.stdout:Install 3 Packages 2023-01-24T17:15:35.682 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:15:35.682 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 85 M 2023-01-24T17:15:35.682 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 94 M 2023-01-24T17:15:35.683 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:15:36.230 INFO:teuthology.orchestra.run.smithi077.stdout:(1/3): kernel-4.18.0-448.el8.x86_64.rpm 17 MB/s | 9.2 MB 00:00 2023-01-24T17:15:37.103 INFO:teuthology.orchestra.run.smithi077.stdout:(2/3): kernel-core-4.18.0-448.el8.x86_64.rpm 29 MB/s | 41 MB 00:01 2023-01-24T17:15:37.405 INFO:teuthology.orchestra.run.smithi077.stdout:(3/3): kernel-modules-4.18.0-448.el8.x86_64.rpm 20 MB/s | 34 MB 00:01 2023-01-24T17:15:37.408 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:15:37.409 INFO:teuthology.orchestra.run.smithi077.stdout:Total 49 MB/s | 85 MB 00:01 2023-01-24T17:15:38.043 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:15:38.436 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:15:38.436 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:15:40.493 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:15:40.519 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:15:43.218 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:15:43.799 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-24T17:15:47.658 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-24T17:15:48.182 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-24T17:15:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-24T17:15:51.758 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-4.18.0-448.el8.x86_64 3/3 2023-01-24T17:16:13.686 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 3/3 2023-01-24T17:16:13.712 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-24T17:16:14.119 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-4.18.0-448.el8.x86_64 3/3 2023-01-24T17:16:14.120 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-4.18.0-448.el8.x86_64 1/3 2023-01-24T17:16:14.120 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-core-4.18.0-448.el8.x86_64 2/3 2023-01-24T17:16:14.702 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-24T17:16:14.703 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:16:14.703 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:16:14.703 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-4.18.0-448.el8.x86_64 kernel-core-4.18.0-448.el8.x86_64 2023-01-24T17:16:14.703 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-4.18.0-448.el8.x86_64 2023-01-24T17:16:14.704 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:16:14.704 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:16:15.624 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel --last | head -n 1 2023-01-24T17:16:15.860 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-4.18.0-448.el8.x86_64 Tue 24 Jan 2023 05:15:51 PM UTC 2023-01-24T17:16:15.861 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-24T17:16:15.862 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-408.el8.x86_64 Expected: 4.18.0-448.el8.x86_64 2023-01-24T17:16:15.862 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-24T17:16:15.862 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-24T17:16:15.862 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-24T17:16:15.863 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=None, version=distro) 2023-01-24T17:16:15.863 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel --last | head -n 1 2023-01-24T17:16:16.052 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-4.18.0-448.el8.x86_64 Tue 24 Jan 2023 05:15:51 PM UTC 2023-01-24T17:16:16.054 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-24T17:16:16.054 DEBUG:teuthology.orchestra.run.smithi077:> sudo rpm -qi grub2-tools 2023-01-24T17:16:16.087 INFO:teuthology.orchestra.run.smithi077.stdout:Name : grub2-tools 2023-01-24T17:16:16.088 INFO:teuthology.orchestra.run.smithi077.stdout:Epoch : 1 2023-01-24T17:16:16.088 INFO:teuthology.orchestra.run.smithi077.stdout:Version : 2.02 2023-01-24T17:16:16.088 INFO:teuthology.orchestra.run.smithi077.stdout:Release : 129.el8 2023-01-24T17:16:16.089 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: x86_64 2023-01-24T17:16:16.089 INFO:teuthology.orchestra.run.smithi077.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-01-24T17:16:16.089 INFO:teuthology.orchestra.run.smithi077.stdout:Group : System Environment/Base 2023-01-24T17:16:16.089 INFO:teuthology.orchestra.run.smithi077.stdout:Size : 9265586 2023-01-24T17:16:16.089 INFO:teuthology.orchestra.run.smithi077.stdout:License : GPLv3+ 2023-01-24T17:16:16.090 INFO:teuthology.orchestra.run.smithi077.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-01-24T17:16:16.090 INFO:teuthology.orchestra.run.smithi077.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-01-24T17:16:16.090 INFO:teuthology.orchestra.run.smithi077.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-01-24T17:16:16.090 INFO:teuthology.orchestra.run.smithi077.stdout:Build Host : kbuilder.bsys.centos.org 2023-01-24T17:16:16.090 INFO:teuthology.orchestra.run.smithi077.stdout:Relocations : (not relocatable) 2023-01-24T17:16:16.090 INFO:teuthology.orchestra.run.smithi077.stdout:Packager : CentOS BuildSystem 2023-01-24T17:16:16.091 INFO:teuthology.orchestra.run.smithi077.stdout:Vendor : CentOS 2023-01-24T17:16:16.091 INFO:teuthology.orchestra.run.smithi077.stdout:URL : http://www.gnu.org/software/grub/ 2023-01-24T17:16:16.091 INFO:teuthology.orchestra.run.smithi077.stdout:Summary : Support tools for GRUB. 2023-01-24T17:16:16.091 INFO:teuthology.orchestra.run.smithi077.stdout:Description : 2023-01-24T17:16:16.091 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:16:16.092 INFO:teuthology.orchestra.run.smithi077.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-01-24T17:16:16.092 INFO:teuthology.orchestra.run.smithi077.stdout:customizable bootloader with modular architecture. It supports a rich 2023-01-24T17:16:16.092 INFO:teuthology.orchestra.run.smithi077.stdout:variety of kernel formats, file systems, computer architectures and 2023-01-24T17:16:16.092 INFO:teuthology.orchestra.run.smithi077.stdout:hardware devices. 2023-01-24T17:16:16.093 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:16:16.093 INFO:teuthology.orchestra.run.smithi077.stdout:This subpackage provides tools for support of all platforms. 2023-01-24T17:16:16.094 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-01-24T17:16:16.094 INFO:teuthology.task.kernel:Updating grub on smithi077 to boot 4.18.0-448.el8.x86_64 2023-01-24T17:16:16.095 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-01-24T17:16:17.320 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2023-01-24T17:16:19.756 INFO:teuthology.orchestra.run.smithi077.stderr:done 2023-01-24T17:16:19.757 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2023-01-24T17:16:19.780 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.W64jE1pyTS 2023-01-24T17:16:19.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.W64jE1pyTS 2023-01-24T17:16:19.868 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.W64jE1pyTS 2023-01-24T17:16:19.979 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.W64jE1pyTS is 6KB 2023-01-24T17:16:20.027 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.W64jE1pyTS 2023-01-24T17:16:20.045 DEBUG:teuthology.orchestra.run.smithi077:> sudo /bin/ls /boot/loader/entries || true 2023-01-24T17:16:20.110 INFO:teuthology.orchestra.run.smithi077.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-01-24T17:16:20.111 INFO:teuthology.orchestra.run.smithi077.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-01-24T17:16:20.111 INFO:teuthology.orchestra.run.smithi077.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-01-24T17:16:20.111 INFO:teuthology.orchestra.run.smithi077.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64.conf 2023-01-24T17:16:20.112 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64 2023-01-24T17:16:20.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2023-01-24T17:16:50.334 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-24T17:16:50.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-01-24T17:16:50.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T17:17:08.844 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2023-01-24T17:17:38.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-01-24T17:17:38.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T17:17:54.420 DEBUG:teuthology.orchestra.run.smithi077:> true 2023-01-24T17:17:54.754 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-01-24T17:17:54.754 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-448.el8.x86_64"... 2023-01-24T17:17:54.754 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2023-01-24T17:17:54.771 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-448.el8.x86_64 2023-01-24T17:17:54.772 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-448.el8.x86_64 vs 4.18.0-448.el8.x86_64 2023-01-24T17:17:54.772 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-24T17:17:54.773 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-01-24T17:17:55.774 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-24T17:17:55.774 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-24T17:17:55.856 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2023-01-24T17:17:55.885 DEBUG:teuthology.parallel:result is None 2023-01-24T17:17:55.885 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-24T17:17:55.894 INFO:teuthology.task.internal:Creating test directory... 2023-01-24T17:17:55.895 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-24T17:17:55.913 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-24T17:17:55.922 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-24T17:17:55.932 INFO:teuthology.task.internal:Creating archive directory... 2023-01-24T17:17:55.933 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-24T17:17:55.977 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-24T17:17:55.988 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-24T17:17:55.989 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-24T17:17:56.054 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T17:17:56.064 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T17:17:56.065 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-24T17:17:56.074 INFO:teuthology.task.internal:Configuring sudo... 2023-01-24T17:17:56.074 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-24T17:17:56.133 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-24T17:17:56.143 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-24T17:17:56.144 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-24T17:17:56.191 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T17:17:56.280 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T17:17:56.319 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T17:17:56.406 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T17:17:56.447 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:17:56.448 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-24T17:17:56.517 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2023-01-24T17:17:56.612 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T17:17:56.859 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-24T17:17:56.872 INFO:teuthology.task.internal:Starting timer... 2023-01-24T17:17:56.872 INFO:teuthology.run_tasks:Running task pcp... 2023-01-24T17:17:56.883 INFO:teuthology.run_tasks:Running task selinux... 2023-01-24T17:17:56.893 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-01-24T17:17:56.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2023-01-24T17:17:57.018 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2023-01-24T17:17:57.020 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-24T17:17:57.020 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2023-01-24T17:17:57.039 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2023-01-24T17:17:57.040 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive'} 2023-01-24T17:17:57.040 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T17:17:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T17:17:57.109 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-24T17:17:57.110 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2023-01-24T17:17:57.181 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-24T17:17:57.193 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-01-24T17:17:57.194 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-24T17:17:57.211 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-24T17:17:57.212 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-24T17:24:27.908 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com')] 2023-01-24T17:24:27.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-01-24T17:24:27.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T17:24:27.979 DEBUG:teuthology.orchestra.run.smithi077:> true 2023-01-24T17:24:28.047 INFO:teuthology.run_tasks:Running task clock... 2023-01-24T17:24:28.058 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-24T17:24:28.058 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-24T17:24:28.059 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T17:24:28.120 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-24T17:24:28.134 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-24T17:24:28.217 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2023-01-24T17:24:28.229 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2023-01-24T17:24:28.242 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-24T17:24:28.256 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-24T17:24:28.320 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2023-01-24T17:24:28.325 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T17:24:28.325 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2023-01-24T17:24:28.326 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T17:24:28.326 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T17:24:28.326 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T17:24:28.326 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T17:24:28.327 INFO:teuthology.run_tasks:Running task install... 2023-01-24T17:24:28.339 DEBUG:teuthology.task.install:project ceph 2023-01-24T17:24:28.340 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}} 2023-01-24T17:24:28.340 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'} 2023-01-24T17:24:28.341 INFO:teuthology.task.install:Using flavor: default 2023-01-24T17:24:28.347 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-24T17:24:28.348 INFO:teuthology.task.install:extra packages: [] 2023-01-24T17:24:28.348 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2023-01-24T17:24:28.402 INFO:teuthology.orchestra.run.smithi077.stdout:CentOSStream 2023-01-24T17:24:28.404 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T17:24:28.405 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'tag': None, 'wait_for_package': False} 2023-01-24T17:24:28.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2023-01-24T17:24:28.481 INFO:teuthology.orchestra.run.smithi077.stdout:CentOSStream 2023-01-24T17:24:28.482 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T17:24:28.483 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T17:24:28.643 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/centos/8/flavors/default/ 2023-01-24T17:24:28.643 INFO:teuthology.task.install.rpm:Package version is 17.2.5-727.g49f8fb05 2023-01-24T17:24:28.772 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-24T17:24:28.772 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:24:28.773 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-24T17:24:28.800 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-01-24T17:24:28.801 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/49f8fb05584886826e8eade75f7105fba754560c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-24T17:24:28.869 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-24T17:24:28.946 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-24T17:24:29.013 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2023-01-24T17:24:29.014 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2023-01-24T17:24:29.263 INFO:teuthology.orchestra.run.smithi077.stdout:61 files removed 2023-01-24T17:24:29.291 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw 2023-01-24T17:24:30.372 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 116 kB/s | 78 kB 00:00 2023-01-24T17:24:30.664 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 36 kB/s | 9.2 kB 00:00 2023-01-24T17:24:30.941 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 5.9 kB/s | 1.5 kB 00:00 2023-01-24T17:24:31.643 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - AppStream 41 MB/s | 27 MB 00:00 2023-01-24T17:24:37.284 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - BaseOS 42 MB/s | 27 MB 00:00 2023-01-24T17:24:41.534 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - Extras 177 kB/s | 18 kB 00:00 2023-01-24T17:24:42.119 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - PowerTools 12 MB/s | 5.6 MB 00:00 2023-01-24T17:24:43.327 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-01-24T17:24:45.144 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 13 MB 00:01 2023-01-24T17:24:48.494 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 413 kB/s | 24 kB 00:00 2023-01-24T17:24:51.892 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:24:51.895 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:24:51.895 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-01-24T17:24:51.895 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:24:51.896 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:24:51.896 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 17 M 2023-01-24T17:24:51.896 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2023-01-24T17:24:51.896 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 3.5 M 2023-01-24T17:24:51.896 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 3.6 M 2023-01-24T17:24:51.897 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-01-24T17:24:51.897 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 5.1 M 2023-01-24T17:24:51.897 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 23 M 2023-01-24T17:24:51.897 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 24 k 2023-01-24T17:24:51.897 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-24T17:24:51.898 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 793 k 2023-01-24T17:24:51.898 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-24T17:24:51.898 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-01-24T17:24:51.898 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 506 k 2023-01-24T17:24:51.898 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-01-24T17:24:51.899 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 5.5 M 2023-01-24T17:24:51.899 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-24T17:24:51.899 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-01-24T17:24:51.899 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 45 k 2023-01-24T17:24:51.899 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 108 k 2023-01-24T17:24:51.899 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 190 k 2023-01-24T17:24:51.900 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-01-24T17:24:51.900 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 370 k 2023-01-24T17:24:51.900 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 357 k 2023-01-24T17:24:51.900 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 107 k 2023-01-24T17:24:51.900 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-24T17:24:51.901 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:24:51.901 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:24:51.902 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:24:51.902 INFO:teuthology.orchestra.run.smithi077.stdout:Install 21 Packages 2023-01-24T17:24:51.902 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 2 Packages 2023-01-24T17:24:51.902 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:24:51.903 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 63 M 2023-01-24T17:24:51.903 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:24:55.745 INFO:teuthology.orchestra.run.smithi077.stdout:(1/23): ceph-base-17.2.5-727.g49f8fb05.el8.x86_ 1.3 MB/s | 5.1 MB 00:03 2023-01-24T17:24:55.845 INFO:teuthology.orchestra.run.smithi077.stdout:(2/23): ceph-selinux-17.2.5-727.g49f8fb05.el8.x 244 kB/s | 24 kB 00:00 2023-01-24T17:24:56.104 INFO:teuthology.orchestra.run.smithi077.stdout:(3/23): libcephfs2-17.2.5-727.g49f8fb05.el8.x86 3.0 MB/s | 793 kB 00:00 2023-01-24T17:24:56.296 INFO:teuthology.orchestra.run.smithi077.stdout:(4/23): libradosstriper1-17.2.5-727.g49f8fb05.e 2.6 MB/s | 506 kB 00:00 2023-01-24T17:24:57.354 INFO:teuthology.orchestra.run.smithi077.stdout:(5/23): librgw2-17.2.5-727.g49f8fb05.el8.x86_64 5.2 MB/s | 5.5 MB 00:01 2023-01-24T17:24:57.471 INFO:teuthology.orchestra.run.smithi077.stdout:(6/23): python3-ceph-argparse-17.2.5-727.g49f8f 383 kB/s | 45 kB 00:00 2023-01-24T17:24:57.588 INFO:teuthology.orchestra.run.smithi077.stdout:(7/23): python3-ceph-common-17.2.5-727.g49f8fb0 925 kB/s | 108 kB 00:00 2023-01-24T17:24:57.755 INFO:teuthology.orchestra.run.smithi077.stdout:(8/23): ceph-radosgw-17.2.5-727.g49f8fb05.el8.x 2.8 MB/s | 17 MB 00:05 2023-01-24T17:24:57.788 INFO:teuthology.orchestra.run.smithi077.stdout:(9/23): python3-cephfs-17.2.5-727.g49f8fb05.el8 951 kB/s | 190 kB 00:00 2023-01-24T17:24:57.889 INFO:teuthology.orchestra.run.smithi077.stdout:(10/23): python3-rados-17.2.5-727.g49f8fb05.el8 2.7 MB/s | 370 kB 00:00 2023-01-24T17:24:57.931 INFO:teuthology.orchestra.run.smithi077.stdout:(11/23): python3-rbd-17.2.5-727.g49f8fb05.el8.x 2.5 MB/s | 357 kB 00:00 2023-01-24T17:24:57.990 INFO:teuthology.orchestra.run.smithi077.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2023-01-24T17:24:58.037 INFO:teuthology.orchestra.run.smithi077.stdout:(13/23): python3-rgw-17.2.5-727.g49f8fb05.el8.x 725 kB/s | 107 kB 00:00 2023-01-24T17:24:58.074 INFO:teuthology.orchestra.run.smithi077.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.0 MB/s | 259 kB 00:00 2023-01-24T17:24:58.100 INFO:teuthology.orchestra.run.smithi077.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.0 MB/s | 47 kB 00:00 2023-01-24T17:24:58.124 INFO:teuthology.orchestra.run.smithi077.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 504 kB/s | 44 kB 00:00 2023-01-24T17:24:58.458 INFO:teuthology.orchestra.run.smithi077.stdout:(17/23): ceph-common-17.2.5-727.g49f8fb05.el8.x 3.5 MB/s | 23 MB 00:06 2023-01-24T17:24:58.514 INFO:teuthology.orchestra.run.smithi077.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 151 kB/s | 59 kB 00:00 2023-01-24T17:24:58.592 INFO:teuthology.orchestra.run.smithi077.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 622 kB/s | 306 kB 00:00 2023-01-24T17:24:58.676 INFO:teuthology.orchestra.run.smithi077.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 346 kB/s | 75 kB 00:00 2023-01-24T17:24:59.101 INFO:teuthology.orchestra.run.smithi077.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 3.0 MB/s | 1.7 MB 00:00 2023-01-24T17:24:59.668 INFO:teuthology.orchestra.run.smithi077.stdout:(22/23): librbd1-17.2.5-727.g49f8fb05.el8.x86_6 3.6 MB/s | 3.6 MB 00:00 2023-01-24T17:25:00.059 INFO:teuthology.orchestra.run.smithi077.stdout:(23/23): librados2-17.2.5-727.g49f8fb05.el8.x86 2.4 MB/s | 3.5 MB 00:01 2023-01-24T17:25:00.063 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:25:00.063 INFO:teuthology.orchestra.run.smithi077.stdout:Total 7.7 MB/s | 63 MB 00:08 2023-01-24T17:25:00.105 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:25:00.150 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:25:00.151 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:25:00.768 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:25:00.772 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:25:01.876 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:25:02.090 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-01-24T17:25:02.294 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-01-24T17:25:02.675 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-01-24T17:25:02.853 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 3/25 2023-01-24T17:25:02.950 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 3/25 2023-01-24T17:25:03.219 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:17.2.5-727.g49f8fb05.el8.x86_64 4/25 2023-01-24T17:25:03.511 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 5/25 2023-01-24T17:25:03.944 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 5/25 2023-01-24T17:25:04.136 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 6/25 2023-01-24T17:25:04.360 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 6/25 2023-01-24T17:25:04.558 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-01-24T17:25:04.717 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-01-24T17:25:04.861 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-24T17:25:05.412 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-24T17:25:05.619 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:17.2.5-727.g49f8fb05.el8.x86_64 10/25 2023-01-24T17:25:05.686 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:17.2.5-727.g49f8fb05.el8.x86_64 10/25 2023-01-24T17:25:05.813 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:17.2.5-727.g49f8fb05.el8.x 11/25 2023-01-24T17:25:06.047 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:17.2.5-727.g49f8fb05.el8.x86_64 12/25 2023-01-24T17:25:06.279 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:17.2.5-727.g49f8fb05.el8.x86_64 13/25 2023-01-24T17:25:06.519 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:17.2.5-727.g49f8fb05.el8.x86_64 14/25 2023-01-24T17:25:06.669 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:17.2.5-727.g49f8fb05.el8.x86_64 15/25 2023-01-24T17:25:06.742 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:17.2.5-727.g49f8fb05.el8.x86_64 15/25 2023-01-24T17:25:06.956 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-01-24T17:25:07.187 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-01-24T17:25:07.383 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-01-24T17:25:07.588 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:17.2.5-727.g49f8fb05.el8.x86 19/25 2023-01-24T17:25:11.421 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 20/25 2023-01-24T17:25:11.677 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 20/25 2023-01-24T17:25:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 20/25 2023-01-24T17:25:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 21/25 2023-01-24T17:25:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 21/25 2023-01-24T17:25:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:17.2.5-727.g49f8fb05.el8.x86_64 22/25 2023-01-24T17:25:32.169 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:17.2.5-727.g49f8fb05.el8.x86_64 22/25 2023-01-24T17:25:32.170 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2023-01-24T17:25:32.170 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2023-01-24T17:25:32.170 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2023-01-24T17:25:32.170 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /run 2023-01-24T17:25:32.171 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2023-01-24T17:25:32.171 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2023-01-24T17:25:32.171 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2023-01-24T17:25:32.171 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2023-01-24T17:25:32.171 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2023-01-24T17:25:32.172 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:25:33.925 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:17.2.5-727.g49f8fb05.el8.x86_64 23/25 2023-01-24T17:25:34.000 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:17.2.5-727.g49f8fb05.el8.x86_64 23/25 2023-01-24T17:25:34.000 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T17:25:34.209 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-24T17:25:34.210 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T17:25:34.936 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T17:25:34.937 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 1/25 2023-01-24T17:25:34.937 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 2/25 2023-01-24T17:25:34.937 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:17.2.5-727.g49f8fb05.el8.x86_64 3/25 2023-01-24T17:25:34.937 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:17.2.5-727.g49f8fb05.el8.x86_64 4/25 2023-01-24T17:25:34.938 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 5/25 2023-01-24T17:25:34.938 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:17.2.5-727.g49f8fb05.el8.x86_64 6/25 2023-01-24T17:25:34.938 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:17.2.5-727.g49f8fb05.el8.x86_64 7/25 2023-01-24T17:25:34.938 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:17.2.5-727.g49f8fb05.el8.x 8/25 2023-01-24T17:25:34.939 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:17.2.5-727.g49f8fb05.el8.x86 9/25 2023-01-24T17:25:34.939 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:17.2.5-727.g49f8fb05.el8.x86_64 10/25 2023-01-24T17:25:34.939 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:17.2.5-727.g49f8fb05.el8.x86_64 11/25 2023-01-24T17:25:34.939 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:17.2.5-727.g49f8fb05.el8.x86_64 12/25 2023-01-24T17:25:34.939 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:17.2.5-727.g49f8fb05.el8.x86_64 13/25 2023-01-24T17:25:34.940 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-01-24T17:25:34.940 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-01-24T17:25:34.940 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-01-24T17:25:34.940 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-01-24T17:25:34.940 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-01-24T17:25:34.941 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-01-24T17:25:34.941 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-01-24T17:25:34.941 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-01-24T17:25:34.941 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 22/25 2023-01-24T17:25:34.941 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-01-24T17:25:34.941 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 24/25 2023-01-24T17:25:35.760 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-01-24T17:25:35.761 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:25:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2023-01-24T17:25:35.761 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.761 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:25:35.762 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.762 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.762 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.762 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.763 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-24T17:25:35.763 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.763 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-24T17:25:35.763 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-24T17:25:35.763 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.763 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-01-24T17:25:35.764 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.764 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-24T17:25:35.764 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-24T17:25:35.764 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.764 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.765 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.765 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-24T17:25:35.765 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.765 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.765 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:35.766 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-24T17:25:35.766 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:25:35.766 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:25:36.050 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-test 2023-01-24T17:25:36.773 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:48 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:25:37.918 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:25:37.921 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:25:37.921 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-01-24T17:25:37.921 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:25:37.921 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:25:37.922 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 52 M 2023-01-24T17:25:37.922 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-01-24T17:25:37.922 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-01-24T17:25:37.922 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 174 k 2023-01-24T17:25:37.922 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-01-24T17:25:37.923 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-01-24T17:25:37.923 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-01-24T17:25:37.923 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:25:37.923 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:25:37.923 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:25:37.923 INFO:teuthology.orchestra.run.smithi077.stdout:Install 6 Packages 2023-01-24T17:25:37.924 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:25:37.924 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 53 M 2023-01-24T17:25:37.924 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 233 M 2023-01-24T17:25:37.925 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:25:37.992 INFO:teuthology.orchestra.run.smithi077.stdout:(1/6): jq-1.6-6.el8.x86_64.rpm 2.8 MB/s | 202 kB 00:00 2023-01-24T17:25:38.042 INFO:teuthology.orchestra.run.smithi077.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-01-24T17:25:38.076 INFO:teuthology.orchestra.run.smithi077.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 9.5 MB/s | 323 kB 00:00 2023-01-24T17:25:38.101 INFO:teuthology.orchestra.run.smithi077.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 71 kB 00:00 2023-01-24T17:25:38.327 INFO:teuthology.orchestra.run.smithi077.stdout:(5/6): libcephsqlite-17.2.5-727.g49f8fb05.el8.x 429 kB/s | 174 kB 00:00 2023-01-24T17:25:42.235 INFO:teuthology.orchestra.run.smithi077.stdout:(6/6): ceph-test-17.2.5-727.g49f8fb05.el8.x86_6 12 MB/s | 52 MB 00:04 2023-01-24T17:25:42.237 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:25:42.238 INFO:teuthology.orchestra.run.smithi077.stdout:Total 12 MB/s | 53 MB 00:04 2023-01-24T17:25:42.281 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:25:42.301 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:25:42.301 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:25:42.630 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:25:42.634 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:25:43.160 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:25:43.387 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-01-24T17:25:43.551 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-01-24T17:25:43.731 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-01-24T17:25:43.805 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-01-24T17:25:44.012 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.6-6.el8.x86_64 4/6 2023-01-24T17:25:44.214 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:17.2.5-727.g49f8fb05.el8.x86_64 5/6 2023-01-24T17:25:49.254 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:17.2.5-727.g49f8fb05.el8.x86_64 5/6 2023-01-24T17:25:49.536 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:17.2.5-727.g49f8fb05.el8.x86_64 6/6 2023-01-24T17:25:49.828 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-test-2:17.2.5-727.g49f8fb05.el8.x86_64 6/6 2023-01-24T17:25:49.828 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:17.2.5-727.g49f8fb05.el8.x86_64 1/6 2023-01-24T17:25:49.829 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:17.2.5-727.g49f8fb05.el8.x86_64 2/6 2023-01-24T17:25:49.829 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.6-6.el8.x86_64 3/6 2023-01-24T17:25:49.829 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-01-24T17:25:49.829 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-01-24T17:25:50.380 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-01-24T17:25:50.381 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:25:50.381 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:25:50.382 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:50.382 INFO:teuthology.orchestra.run.smithi077.stdout: jq-1.6-6.el8.x86_64 2023-01-24T17:25:50.382 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:25:50.382 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-24T17:25:50.383 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-1.el8.x86_64 2023-01-24T17:25:50.383 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-24T17:25:50.383 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:25:50.383 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:25:50.558 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph 2023-01-24T17:25:51.288 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:03 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:25:52.423 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:25:52.427 INFO:teuthology.orchestra.run.smithi077.stdout:====================================================================================== 2023-01-24T17:25:52.427 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-01-24T17:25:52.427 INFO:teuthology.orchestra.run.smithi077.stdout:====================================================================================== 2023-01-24T17:25:52.427 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:25:52.428 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 6.4 k 2023-01-24T17:25:52.428 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-01-24T17:25:52.428 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 2.3 M 2023-01-24T17:25:52.428 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 1.6 M 2023-01-24T17:25:52.428 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 233 k 2023-01-24T17:25:52.429 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 3.9 M 2023-01-24T17:25:52.429 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 17 M 2023-01-24T17:25:52.429 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-24T17:25:52.429 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-24T17:25:52.429 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-01-24T17:25:52.430 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-24T17:25:52.430 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-24T17:25:52.430 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-01-24T17:25:52.431 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-24T17:25:52.432 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-24T17:25:52.432 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-24T17:25:52.432 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2023-01-24T17:25:52.433 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-01-24T17:25:52.433 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-24T17:25:52.434 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-24T17:25:52.434 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-24T17:25:52.434 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-01-24T17:25:52.435 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-01-24T17:25:52.435 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-01-24T17:25:52.436 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-24T17:25:52.436 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-24T17:25:52.436 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-24T17:25:52.436 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-24T17:25:52.437 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-24T17:25:52.437 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-24T17:25:52.437 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-24T17:25:52.437 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-01-24T17:25:52.438 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-24T17:25:52.438 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:25:52.438 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:25:52.438 INFO:teuthology.orchestra.run.smithi077.stdout:====================================================================================== 2023-01-24T17:25:52.438 INFO:teuthology.orchestra.run.smithi077.stdout:Install 32 Packages 2023-01-24T17:25:52.438 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:25:52.439 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 28 M 2023-01-24T17:25:52.440 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 110 M 2023-01-24T17:25:52.440 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:25:52.579 INFO:teuthology.orchestra.run.smithi077.stdout:(1/32): ceph-17.2.5-727.g49f8fb05.el8.x86_64.rp 44 kB/s | 6.4 kB 00:00 2023-01-24T17:25:53.496 INFO:teuthology.orchestra.run.smithi077.stdout:(2/32): ceph-mgr-17.2.5-727.g49f8fb05.el8.x86_6 1.5 MB/s | 1.6 MB 00:01 2023-01-24T17:25:53.620 INFO:teuthology.orchestra.run.smithi077.stdout:(3/32): ceph-mds-17.2.5-727.g49f8fb05.el8.x86_6 1.9 MB/s | 2.3 MB 00:01 2023-01-24T17:25:53.696 INFO:teuthology.orchestra.run.smithi077.stdout:(4/32): ceph-mgr-modules-core-17.2.5-727.g49f8f 3.0 MB/s | 233 kB 00:00 2023-01-24T17:25:53.755 INFO:teuthology.orchestra.run.smithi077.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-01-24T17:25:53.772 INFO:teuthology.orchestra.run.smithi077.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 2.3 MB/s | 39 kB 00:00 2023-01-24T17:25:53.797 INFO:teuthology.orchestra.run.smithi077.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.0 MB/s | 103 kB 00:00 2023-01-24T17:25:53.847 INFO:teuthology.orchestra.run.smithi077.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2023-01-24T17:25:53.881 INFO:teuthology.orchestra.run.smithi077.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2023-01-24T17:25:53.906 INFO:teuthology.orchestra.run.smithi077.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.2 MB/s | 237 kB 00:00 2023-01-24T17:25:53.940 INFO:teuthology.orchestra.run.smithi077.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 16 MB/s | 559 kB 00:00 2023-01-24T17:25:53.965 INFO:teuthology.orchestra.run.smithi077.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2023-01-24T17:25:54.124 INFO:teuthology.orchestra.run.smithi077.stdout:(13/32): ceph-mon-17.2.5-727.g49f8fb05.el8.x86_ 2.5 MB/s | 3.9 MB 00:01 2023-01-24T17:25:54.149 INFO:teuthology.orchestra.run.smithi077.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 238 kB/s | 44 kB 00:00 2023-01-24T17:25:54.233 INFO:teuthology.orchestra.run.smithi077.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.7 MB/s | 185 kB 00:00 2023-01-24T17:25:54.316 INFO:teuthology.orchestra.run.smithi077.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.0 MB/s | 173 kB 00:00 2023-01-24T17:25:54.342 INFO:teuthology.orchestra.run.smithi077.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.5 MB/s | 384 kB 00:00 2023-01-24T17:25:54.367 INFO:teuthology.orchestra.run.smithi077.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-01-24T17:25:54.392 INFO:teuthology.orchestra.run.smithi077.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 350 kB/s | 18 kB 00:00 2023-01-24T17:25:54.426 INFO:teuthology.orchestra.run.smithi077.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 830 kB/s | 49 kB 00:00 2023-01-24T17:25:54.451 INFO:teuthology.orchestra.run.smithi077.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2023-01-24T17:25:54.501 INFO:teuthology.orchestra.run.smithi077.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2023-01-24T17:25:54.527 INFO:teuthology.orchestra.run.smithi077.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2023-01-24T17:25:54.552 INFO:teuthology.orchestra.run.smithi077.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 372 kB/s | 19 kB 00:00 2023-01-24T17:25:54.578 INFO:teuthology.orchestra.run.smithi077.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 474 kB/s | 24 kB 00:00 2023-01-24T17:25:54.611 INFO:teuthology.orchestra.run.smithi077.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 498 kB/s | 29 kB 00:00 2023-01-24T17:25:54.636 INFO:teuthology.orchestra.run.smithi077.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 470 kB/s | 27 kB 00:00 2023-01-24T17:25:54.704 INFO:teuthology.orchestra.run.smithi077.stdout:(28/32): python3-waitress-1.2.1-2.el8.1.noarch. 2.5 MB/s | 241 kB 00:00 2023-01-24T17:25:54.754 INFO:teuthology.orchestra.run.smithi077.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2023-01-24T17:25:54.779 INFO:teuthology.orchestra.run.smithi077.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-01-24T17:25:54.804 INFO:teuthology.orchestra.run.smithi077.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 456 kB/s | 23 kB 00:00 2023-01-24T17:25:55.146 INFO:teuthology.orchestra.run.smithi077.stdout:(32/32): ceph-osd-17.2.5-727.g49f8fb05.el8.x86_ 10 MB/s | 17 MB 00:01 2023-01-24T17:25:55.149 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:25:55.149 INFO:teuthology.orchestra.run.smithi077.stdout:Total 10 MB/s | 28 MB 00:02 2023-01-24T17:25:55.226 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:25:55.262 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:25:55.263 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:25:55.629 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:25:55.632 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:25:56.961 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:25:57.243 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-01-24T17:25:57.479 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-01-24T17:25:57.767 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-01-24T17:25:58.069 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2023-01-24T17:25:58.241 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-01-24T17:25:58.404 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-01-24T17:25:58.623 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-01-24T17:25:58.809 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-01-24T17:25:59.049 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-01-24T17:25:59.296 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-01-24T17:25:59.567 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-01-24T17:25:59.797 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-01-24T17:26:00.194 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-01-24T17:26:00.473 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-01-24T17:26:00.662 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-01-24T17:26:00.894 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-01-24T17:26:01.121 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-24T17:26:01.445 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-01-24T17:26:01.692 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-01-24T17:26:01.922 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-01-24T17:26:02.142 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-01-24T17:26:02.869 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-01-24T17:26:03.267 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-01-24T17:26:03.534 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-01-24T17:26:03.889 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2023-01-24T17:26:04.241 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-01-24T17:26:04.667 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:17.2.5-727.g49f8fb05.el8.n 27/32 2023-01-24T17:26:04.896 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 28/32 2023-01-24T17:26:06.378 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 28/32 2023-01-24T17:26:06.557 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:17.2.5-727.g49f8fb05.el8.x86_64 29/32 2023-01-24T17:26:06.950 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:17.2.5-727.g49f8fb05.el8.x86_64 29/32 2023-01-24T17:26:07.162 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:17.2.5-727.g49f8fb05.el8.x86_64 30/32 2023-01-24T17:26:07.377 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:17.2.5-727.g49f8fb05.el8.x86_64 30/32 2023-01-24T17:26:07.528 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:17.2.5-727.g49f8fb05.el8.x86_64 31/32 2023-01-24T17:26:07.546 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:17.2.5-727.g49f8fb05.el8.x86_64 31/32 2023-01-24T17:26:07.692 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:17.2.5-727.g49f8fb05.el8.x86_64 32/32 2023-01-24T17:26:08.141 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-2:17.2.5-727.g49f8fb05.el8.x86_64 32/32 2023-01-24T17:26:08.142 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:17.2.5-727.g49f8fb05.el8.x86_64 1/32 2023-01-24T17:26:08.142 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:17.2.5-727.g49f8fb05.el8.x86_64 2/32 2023-01-24T17:26:08.142 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 3/32 2023-01-24T17:26:08.142 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:17.2.5-727.g49f8fb05.el8.x86_64 4/32 2023-01-24T17:26:08.142 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:17.2.5-727.g49f8fb05.el8.x86_64 5/32 2023-01-24T17:26:08.143 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:17.2.5-727.g49f8fb05.el8.n 6/32 2023-01-24T17:26:08.143 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2023-01-24T17:26:08.143 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-01-24T17:26:08.143 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-01-24T17:26:08.143 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-01-24T17:26:08.144 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-01-24T17:26:08.144 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-01-24T17:26:08.144 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-01-24T17:26:08.144 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-01-24T17:26:08.144 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-01-24T17:26:08.145 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-01-24T17:26:08.145 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-24T17:26:08.145 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-01-24T17:26:08.145 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-01-24T17:26:08.146 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-01-24T17:26:08.146 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-01-24T17:26:08.146 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-01-24T17:26:08.146 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-01-24T17:26:08.146 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-01-24T17:26:08.146 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-01-24T17:26:08.147 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-01-24T17:26:08.148 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-01-24T17:26:08.148 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-01-24T17:26:08.148 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2023-01-24T17:26:08.148 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-01-24T17:26:08.149 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-01-24T17:26:09.352 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-01-24T17:26:09.352 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:09.352 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:26:09.353 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:26:09.353 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:26:09.353 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:26:09.353 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:09.353 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:26:09.354 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:26:09.354 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-24T17:26:09.355 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-24T17:26:09.355 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-01-24T17:26:09.355 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-24T17:26:09.356 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-24T17:26:09.356 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-01-24T17:26:09.356 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-24T17:26:09.356 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-24T17:26:09.357 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-24T17:26:09.357 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-24T17:26:09.357 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-24T17:26:09.357 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-24T17:26:09.358 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-24T17:26:09.359 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-2.6-1.el8.noarch 2023-01-24T17:26:09.359 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-01-24T17:26:09.360 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.14-14.el8.noarch 2023-01-24T17:26:09.360 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-24T17:26:09.360 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-24T17:26:09.360 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-24T17:26:09.361 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-24T17:26:09.361 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-24T17:26:09.361 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-24T17:26:09.361 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-24T17:26:09.361 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-24T17:26:09.362 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-24T17:26:09.362 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-24T17:26:09.362 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:09.362 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:26:09.548 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-base 2023-01-24T17:26:10.264 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:22 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:26:11.301 INFO:teuthology.orchestra.run.smithi077.stdout:Package ceph-base-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:26:11.399 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:26:11.400 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-01-24T17:26:11.401 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:26:11.481 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install cephadm 2023-01-24T17:26:12.193 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:24 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:26:13.341 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:26:13.342 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:13.342 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-01-24T17:26:13.343 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:13.343 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:26:13.343 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 78 k 2023-01-24T17:26:13.343 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:13.343 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:26:13.344 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:13.344 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-01-24T17:26:13.344 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:13.344 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 78 k 2023-01-24T17:26:13.344 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 355 k 2023-01-24T17:26:13.345 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:26:13.834 INFO:teuthology.orchestra.run.smithi077.stdout:cephadm-17.2.5-727.g49f8fb05.el8.noarch.rpm 159 kB/s | 78 kB 00:00 2023-01-24T17:26:13.835 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:26:13.835 INFO:teuthology.orchestra.run.smithi077.stdout:Total 159 kB/s | 78 kB 00:00 2023-01-24T17:26:13.836 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:26:13.844 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:26:13.844 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:26:13.845 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:26:13.849 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:26:13.898 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:26:15.474 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:26:15.656 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:26:15.896 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:26:16.378 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:26:16.378 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:16.379 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:26:16.379 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:16.379 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:16.380 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:26:16.706 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-immutable-object-cache 2023-01-24T17:26:17.432 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:29 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:26:18.582 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:26:18.583 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:18.583 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2023-01-24T17:26:18.583 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:18.583 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:26:18.583 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 162 k 2023-01-24T17:26:18.584 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:18.584 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:26:18.584 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:18.584 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-01-24T17:26:18.585 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:18.585 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 162 k 2023-01-24T17:26:18.585 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 490 k 2023-01-24T17:26:18.586 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:26:19.201 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-immutable-object-cache-17.2.5-727.g49f8fb0 262 kB/s | 162 kB 00:00 2023-01-24T17:26:19.202 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:26:19.202 INFO:teuthology.orchestra.run.smithi077.stdout:Total 261 kB/s | 162 kB 00:00 2023-01-24T17:26:19.203 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:26:19.213 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:26:19.213 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:26:19.263 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:26:19.266 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:26:20.058 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:26:21.319 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:17.2.5-727.g49f8fb05.e 1/1 2023-01-24T17:26:21.763 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.5-727.g49f8fb05.e 1/1 2023-01-24T17:26:22.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:17.2.5-727.g49f8fb05.e 1/1 2023-01-24T17:26:22.253 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:22.254 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:26:22.254 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:26:22.254 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:22.254 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:26:22.463 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr 2023-01-24T17:26:23.194 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:35 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:26:24.240 INFO:teuthology.orchestra.run.smithi077.stdout:Package ceph-mgr-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:26:24.339 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:26:24.340 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-01-24T17:26:24.340 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:26:24.420 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-dashboard 2023-01-24T17:26:25.142 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:37 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:26:26.277 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:26:26.278 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:26.278 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-01-24T17:26:26.278 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:26.279 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:26:26.279 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 1.6 M 2023-01-24T17:26:26.279 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-01-24T17:26:26.279 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 23 k 2023-01-24T17:26:26.279 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 14 k 2023-01-24T17:26:26.280 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-01-24T17:26:26.280 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-24T17:26:26.280 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-24T17:26:26.280 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:26.280 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:26:26.281 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:26.281 INFO:teuthology.orchestra.run.smithi077.stdout:Install 6 Packages 2023-01-24T17:26:26.281 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:26.282 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 2.0 M 2023-01-24T17:26:26.282 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 7.8 M 2023-01-24T17:26:26.282 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:26:26.437 INFO:teuthology.orchestra.run.smithi077.stdout:(1/6): ceph-grafana-dashboards-17.2.5-727.g49f8 147 kB/s | 23 kB 00:00 2023-01-24T17:26:26.454 INFO:teuthology.orchestra.run.smithi077.stdout:(2/6): ceph-prometheus-alerts-17.2.5-727.g49f8f 83 kB/s | 14 kB 00:00 2023-01-24T17:26:26.496 INFO:teuthology.orchestra.run.smithi077.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 728 kB/s | 43 kB 00:00 2023-01-24T17:26:26.572 INFO:teuthology.orchestra.run.smithi077.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 287 kB/s | 34 kB 00:00 2023-01-24T17:26:26.655 INFO:teuthology.orchestra.run.smithi077.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2023-01-24T17:26:26.806 INFO:teuthology.orchestra.run.smithi077.stdout:(6/6): ceph-mgr-dashboard-17.2.5-727.g49f8fb05. 3.1 MB/s | 1.6 MB 00:00 2023-01-24T17:26:26.807 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:26:26.808 INFO:teuthology.orchestra.run.smithi077.stdout:Total 3.7 MB/s | 2.0 MB 00:00 2023-01-24T17:26:26.818 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:26:26.826 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:26:26.827 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:26:26.907 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:26:26.910 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:26:27.222 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:26:27.431 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-01-24T17:26:27.668 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-01-24T17:26:27.862 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-01-24T17:26:28.007 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:17.2.5-727.g49f8fb05.el8.no 4/6 2023-01-24T17:26:28.372 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:17.2.5-727.g49f8fb05.el8.n 5/6 2023-01-24T17:26:28.604 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:17.2.5-727.g49f8fb05.el8.noarch 6/6 2023-01-24T17:26:28.902 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.5-727.g49f8fb05.el8.noarch 6/6 2023-01-24T17:26:28.902 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:17.2.5-727.g49f8fb05.el8.n 1/6 2023-01-24T17:26:28.903 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:17.2.5-727.g49f8fb05.el8.noarch 2/6 2023-01-24T17:26:28.903 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:17.2.5-727.g49f8fb05.el8.no 3/6 2023-01-24T17:26:28.903 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-01-24T17:26:28.903 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-01-24T17:26:29.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-01-24T17:26:29.390 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:29.390 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:26:29.391 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:29.391 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:29.391 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:29.391 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-24T17:26:29.391 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-24T17:26:29.392 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-24T17:26:29.392 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:29.392 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:26:29.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-diskprediction-local 2023-01-24T17:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:42 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:26:31.346 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:26:31.347 INFO:teuthology.orchestra.run.smithi077.stdout:===================================================================================================== 2023-01-24T17:26:31.347 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-01-24T17:26:31.347 INFO:teuthology.orchestra.run.smithi077.stdout:===================================================================================================== 2023-01-24T17:26:31.347 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:26:31.348 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 7.4 M 2023-01-24T17:26:31.348 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-01-24T17:26:31.348 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-01-24T17:26:31.349 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-01-24T17:26:31.349 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-01-24T17:26:31.349 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-01-24T17:26:31.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-01-24T17:26:31.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-01-24T17:26:31.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-01-24T17:26:31.350 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:31.350 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:26:31.350 INFO:teuthology.orchestra.run.smithi077.stdout:===================================================================================================== 2023-01-24T17:26:31.350 INFO:teuthology.orchestra.run.smithi077.stdout:Install 8 Packages 2023-01-24T17:26:31.350 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:31.351 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 35 M 2023-01-24T17:26:31.351 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 210 M 2023-01-24T17:26:31.352 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:26:31.666 INFO:teuthology.orchestra.run.smithi077.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 15 MB/s | 4.9 MB 00:00 2023-01-24T17:26:31.738 INFO:teuthology.orchestra.run.smithi077.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2023-01-24T17:26:31.772 INFO:teuthology.orchestra.run.smithi077.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2023-01-24T17:26:31.872 INFO:teuthology.orchestra.run.smithi077.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 18 MB/s | 3.7 MB 00:00 2023-01-24T17:26:31.914 INFO:teuthology.orchestra.run.smithi077.stdout:(5/8): libgfortran-8.5.0-18.el8.x86_64.rpm 15 MB/s | 645 kB 00:00 2023-01-24T17:26:31.947 INFO:teuthology.orchestra.run.smithi077.stdout:(6/8): libquadmath-8.5.0-18.el8.x86_64.rpm 5.0 MB/s | 172 kB 00:00 2023-01-24T17:26:32.206 INFO:teuthology.orchestra.run.smithi077.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 32 MB/s | 14 MB 00:00 2023-01-24T17:26:33.248 INFO:teuthology.orchestra.run.smithi077.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.5-727 3.9 MB/s | 7.4 MB 00:01 2023-01-24T17:26:33.250 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:26:33.250 INFO:teuthology.orchestra.run.smithi077.stdout:Total 19 MB/s | 35 MB 00:01 2023-01-24T17:26:33.483 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:26:33.501 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:26:33.501 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:26:33.891 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:26:33.894 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:26:34.512 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:26:34.658 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-24T17:26:34.795 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-24T17:26:34.968 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-24T17:26:35.564 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-24T17:26:35.741 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-01-24T17:26:36.357 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-01-24T17:26:36.796 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-24T17:26:37.925 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-24T17:26:38.319 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-01-24T17:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-24T17:26:40.303 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-24T17:26:41.587 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-01-24T17:26:41.790 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.5-727.g49f8fb05 8/8 2023-01-24T17:26:42.123 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.5-727.g49f8fb05 8/8 2023-01-24T17:26:42.123 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.5-727.g49f8fb05 1/8 2023-01-24T17:26:42.123 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-01-24T17:26:42.123 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-01-24T17:26:42.124 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-01-24T17:26:42.124 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-01-24T17:26:42.124 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-01-24T17:26:42.124 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 7/8 2023-01-24T17:26:42.617 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 8/8 2023-01-24T17:26:42.617 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:42.617 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:26:42.618 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:42.618 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-01-24T17:26:42.618 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-01-24T17:26:42.618 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.15-4.el8.x86_64 2023-01-24T17:26:42.618 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-01-24T17:26:42.619 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-01-24T17:26:42.619 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-01-24T17:26:42.619 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-01-24T17:26:42.619 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:42.619 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:26:42.729 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-rook 2023-01-24T17:26:43.449 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:55 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:26:44.591 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:26:44.593 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2023-01-24T17:26:44.593 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-01-24T17:26:44.593 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2023-01-24T17:26:44.593 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:26:44.593 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 47 k 2023-01-24T17:26:44.594 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-01-24T17:26:44.594 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-24T17:26:44.594 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-24T17:26:44.594 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-24T17:26:44.594 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-01-24T17:26:44.595 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-01-24T17:26:44.595 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-24T17:26:44.595 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-01-24T17:26:44.595 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-01-24T17:26:44.595 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-01-24T17:26:44.596 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-01-24T17:26:44.596 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-24T17:26:44.596 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-24T17:26:44.596 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:44.596 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:26:44.597 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2023-01-24T17:26:44.597 INFO:teuthology.orchestra.run.smithi077.stdout:Install 13 Packages 2023-01-24T17:26:44.597 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:44.598 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 1.9 M 2023-01-24T17:26:44.598 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 25 M 2023-01-24T17:26:44.598 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:26:44.671 INFO:teuthology.orchestra.run.smithi077.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 357 kB/s | 27 kB 00:00 2023-01-24T17:26:44.696 INFO:teuthology.orchestra.run.smithi077.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 200 kB/s | 20 kB 00:00 2023-01-24T17:26:44.749 INFO:teuthology.orchestra.run.smithi077.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.7 MB/s | 126 kB 00:00 2023-01-24T17:26:44.780 INFO:teuthology.orchestra.run.smithi077.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.3 MB/s | 110 kB 00:00 2023-01-24T17:26:44.797 INFO:teuthology.orchestra.run.smithi077.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.2 MB/s | 155 kB 00:00 2023-01-24T17:26:44.823 INFO:teuthology.orchestra.run.smithi077.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2023-01-24T17:26:44.922 INFO:teuthology.orchestra.run.smithi077.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 163 kB/s | 16 kB 00:00 2023-01-24T17:26:44.948 INFO:teuthology.orchestra.run.smithi077.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 219 kB/s | 33 kB 00:00 2023-01-24T17:26:45.015 INFO:teuthology.orchestra.run.smithi077.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 882 kB/s | 81 kB 00:00 2023-01-24T17:26:45.057 INFO:teuthology.orchestra.run.smithi077.stdout:(10/13): ceph-mgr-rook-17.2.5-727.g49f8fb05.el8 102 kB/s | 47 kB 00:00 2023-01-24T17:26:45.082 INFO:teuthology.orchestra.run.smithi077.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 917 kB/s | 61 kB 00:00 2023-01-24T17:26:45.220 INFO:teuthology.orchestra.run.smithi077.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 4.3 MB/s | 1.2 MB 00:00 2023-01-24T17:26:45.249 INFO:teuthology.orchestra.run.smithi077.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 318 kB/s | 61 kB 00:00 2023-01-24T17:26:45.251 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:26:45.251 INFO:teuthology.orchestra.run.smithi077.stdout:Total 2.9 MB/s | 1.9 MB 00:00 2023-01-24T17:26:45.303 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:26:45.318 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:26:45.318 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:26:45.414 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:26:45.418 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:26:45.937 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:26:46.250 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-01-24T17:26:46.448 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-01-24T17:26:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-01-24T17:26:46.905 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-01-24T17:26:47.061 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-01-24T17:26:47.310 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-01-24T17:26:47.651 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-01-24T17:26:47.893 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-01-24T17:26:48.993 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-01-24T17:26:49.271 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-01-24T17:26:49.504 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-01-24T17:26:49.698 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-01-24T17:26:49.839 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:17.2.5-727.g49f8fb05.el8.noarch 13/13 2023-01-24T17:26:50.158 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.5-727.g49f8fb05.el8.noarch 13/13 2023-01-24T17:26:50.158 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:17.2.5-727.g49f8fb05.el8.noarch 1/13 2023-01-24T17:26:50.158 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-01-24T17:26:50.159 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-01-24T17:26:50.159 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-01-24T17:26:50.159 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-01-24T17:26:50.159 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-01-24T17:26:50.159 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-01-24T17:26:50.160 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-01-24T17:26:50.160 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-01-24T17:26:50.160 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-01-24T17:26:50.160 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-01-24T17:26:50.160 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-01-24T17:26:50.706 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-01-24T17:26:50.707 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:50.707 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:26:50.707 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:50.707 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-24T17:26:50.708 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-24T17:26:50.708 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-24T17:26:50.708 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-24T17:26:50.708 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-24T17:26:50.708 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-24T17:26:50.709 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-24T17:26:50.709 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-24T17:26:50.709 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-24T17:26:50.709 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-24T17:26:50.709 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-24T17:26:50.709 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-24T17:26:50.710 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:50.710 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:26:50.852 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-cephadm 2023-01-24T17:26:51.574 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:03 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:26:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:26:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:52.719 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-01-24T17:26:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:26:52.720 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 115 k 2023-01-24T17:26:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-01-24T17:26:52.720 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-24T17:26:52.721 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-01-24T17:26:52.721 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-01-24T17:26:52.721 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-24T17:26:52.721 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:52.722 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:26:52.722 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:52.722 INFO:teuthology.orchestra.run.smithi077.stdout:Install 5 Packages 2023-01-24T17:26:52.722 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:52.723 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 5.9 M 2023-01-24T17:26:52.723 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 26 M 2023-01-24T17:26:52.723 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:26:52.806 INFO:teuthology.orchestra.run.smithi077.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.3 MB/s | 538 kB 00:00 2023-01-24T17:26:52.906 INFO:teuthology.orchestra.run.smithi077.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 26 MB/s | 4.8 MB 00:00 2023-01-24T17:26:53.065 INFO:teuthology.orchestra.run.smithi077.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 382 kB/s | 60 kB 00:00 2023-01-24T17:26:53.132 INFO:teuthology.orchestra.run.smithi077.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.4 MB/s | 476 kB 00:00 2023-01-24T17:26:53.307 INFO:teuthology.orchestra.run.smithi077.stdout:(5/5): ceph-mgr-cephadm-17.2.5-727.g49f8fb05.el 197 kB/s | 115 kB 00:00 2023-01-24T17:26:53.308 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:26:53.309 INFO:teuthology.orchestra.run.smithi077.stdout:Total 10 MB/s | 5.9 MB 00:00 2023-01-24T17:26:53.360 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:26:53.371 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:26:53.372 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:26:53.507 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:26:53.511 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:26:53.821 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:26:54.169 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-01-24T17:26:55.111 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-01-24T17:26:55.458 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-01-24T17:26:55.706 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-01-24T17:26:55.922 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 5/5 2023-01-24T17:26:56.230 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 5/5 2023-01-24T17:26:56.230 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 1/5 2023-01-24T17:26:56.231 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-01-24T17:26:56.231 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-01-24T17:26:56.231 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-01-24T17:26:56.713 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-01-24T17:26:56.713 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:56.714 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:26:56.714 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:26:56.714 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-24T17:26:56.714 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.5.1-7.el8.noarch 2023-01-24T17:26:56.714 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-01-24T17:26:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-24T17:26:56.715 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:56.715 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:26:56.842 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-fuse 2023-01-24T17:26:57.570 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:09 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:26:58.708 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:26:58.709 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:58.709 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-01-24T17:26:58.709 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:58.709 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:26:58.710 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 872 k 2023-01-24T17:26:58.710 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-01-24T17:26:58.710 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-01-24T17:26:58.710 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:58.710 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:26:58.711 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:26:58.711 INFO:teuthology.orchestra.run.smithi077.stdout:Install 2 Packages 2023-01-24T17:26:58.711 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:26:58.711 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 954 k 2023-01-24T17:26:58.711 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 3.0 M 2023-01-24T17:26:58.712 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:26:58.789 INFO:teuthology.orchestra.run.smithi077.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.0 MB/s | 83 kB 00:00 2023-01-24T17:26:59.465 INFO:teuthology.orchestra.run.smithi077.stdout:(2/2): ceph-fuse-17.2.5-727.g49f8fb05.el8.x86_6 1.1 MB/s | 872 kB 00:00 2023-01-24T17:26:59.466 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:26:59.466 INFO:teuthology.orchestra.run.smithi077.stdout:Total 1.2 MB/s | 954 kB 00:00 2023-01-24T17:26:59.477 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:26:59.497 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:26:59.498 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:26:59.593 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:26:59.597 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:26:59.779 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:27:00.014 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2023-01-24T17:27:00.236 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 2/2 2023-01-24T17:27:00.644 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 2/2 2023-01-24T17:27:00.645 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 1/2 2023-01-24T17:27:01.156 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2023-01-24T17:27:01.157 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:01.157 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:27:01.157 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2023-01-24T17:27:01.157 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:01.158 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:01.357 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-volume 2023-01-24T17:27:02.098 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:14 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:03.247 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:03.248 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:03.249 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-01-24T17:27:03.249 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:03.249 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:27:03.249 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:17.2.5-727.g49f8fb05.el8 ceph-noarch 263 k 2023-01-24T17:27:03.250 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:03.250 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:27:03.250 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:03.250 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-01-24T17:27:03.251 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:03.251 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 263 k 2023-01-24T17:27:03.251 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 1.3 M 2023-01-24T17:27:03.251 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:27:03.902 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-volume-17.2.5-727.g49f8fb05.el8.noarch.rpm 403 kB/s | 263 kB 00:00 2023-01-24T17:27:03.902 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:27:03.903 INFO:teuthology.orchestra.run.smithi077.stdout:Total 402 kB/s | 263 kB 00:00 2023-01-24T17:27:03.903 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:27:03.913 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:27:03.913 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:27:04.020 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:27:04.023 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:27:04.210 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:27:04.471 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:27:04.895 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:27:05.402 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:17.2.5-727.g49f8fb05.el8.noarch 1/1 2023-01-24T17:27:05.402 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:05.402 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:27:05.403 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:17.2.5-727.g49f8fb05.el8.noarch 2023-01-24T17:27:05.403 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:05.403 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:05.580 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librados-devel 2023-01-24T17:27:06.318 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:18 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:07.453 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:07.454 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:07.454 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2023-01-24T17:27:07.455 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:07.455 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:27:07.455 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 130 k 2023-01-24T17:27:07.455 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:07.455 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:27:07.456 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:07.456 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-01-24T17:27:07.456 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:07.456 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 130 k 2023-01-24T17:27:07.456 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 464 k 2023-01-24T17:27:07.457 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:27:07.949 INFO:teuthology.orchestra.run.smithi077.stdout:librados-devel-17.2.5-727.g49f8fb05.el8.x86_64. 263 kB/s | 130 kB 00:00 2023-01-24T17:27:07.950 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:27:07.950 INFO:teuthology.orchestra.run.smithi077.stdout:Total 262 kB/s | 130 kB 00:00 2023-01-24T17:27:07.950 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:27:07.955 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:27:07.955 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:27:08.006 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:27:08.009 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:27:08.144 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:27:08.309 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:08.599 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:09.104 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:09.104 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:09.105 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:27:09.105 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:27:09.105 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:09.105 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:09.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install libcephfs2 2023-01-24T17:27:10.066 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:22 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:11.096 INFO:teuthology.orchestra.run.smithi077.stdout:Package libcephfs2-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:11.196 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:11.197 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-01-24T17:27:11.197 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:11.616 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install libcephfs-devel 2023-01-24T17:27:12.349 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:24 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:13.481 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:13.482 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:13.482 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2023-01-24T17:27:13.482 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:13.483 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:27:13.483 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 25 k 2023-01-24T17:27:13.483 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:13.483 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:27:13.484 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:13.484 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-01-24T17:27:13.484 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:13.485 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 25 k 2023-01-24T17:27:13.485 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 109 k 2023-01-24T17:27:13.485 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:27:13.940 INFO:teuthology.orchestra.run.smithi077.stdout:libcephfs-devel-17.2.5-727.g49f8fb05.el8.x86_64 54 kB/s | 25 kB 00:00 2023-01-24T17:27:13.941 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:27:13.941 INFO:teuthology.orchestra.run.smithi077.stdout:Total 54 kB/s | 25 kB 00:00 2023-01-24T17:27:13.941 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:27:13.944 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:27:13.945 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:27:13.946 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:27:13.950 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:27:14.006 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:27:14.175 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:14.447 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:14.893 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:14.894 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:14.894 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:27:14.894 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:27:14.894 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:14.895 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:14.998 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librados2 2023-01-24T17:27:15.736 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:27 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:16.778 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:16.877 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:16.878 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-01-24T17:27:16.879 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:16.967 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librbd1 2023-01-24T17:27:17.700 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:29 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:18.743 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:18.842 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:18.843 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-01-24T17:27:18.843 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:18.935 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rados 2023-01-24T17:27:19.655 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:31 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:20.693 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rados-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:20.793 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:20.794 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-01-24T17:27:20.794 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:20.884 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rgw 2023-01-24T17:27:21.605 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:33 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:22.660 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rgw-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:22.762 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:22.764 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-01-24T17:27:22.764 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:22.951 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-cephfs 2023-01-24T17:27:23.671 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:35 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:24.715 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-cephfs-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:24.814 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:24.815 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-01-24T17:27:24.816 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:24.892 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rbd 2023-01-24T17:27:25.616 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:37 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:26.654 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rbd-2:17.2.5-727.g49f8fb05.el8.x86_64 is already installed. 2023-01-24T17:27:26.754 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:26.755 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-01-24T17:27:26.756 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:26.841 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-fuse 2023-01-24T17:27:27.565 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:39 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:28.701 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:28.702 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:28.702 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2023-01-24T17:27:28.703 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:28.703 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:27:28.703 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 90 k 2023-01-24T17:27:28.703 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:28.703 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:27:28.703 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:28.704 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-01-24T17:27:28.704 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:28.705 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 90 k 2023-01-24T17:27:28.705 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 264 k 2023-01-24T17:27:28.705 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:27:29.237 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-fuse-17.2.5-727.g49f8fb05.el8.x86_64.rpm 169 kB/s | 90 kB 00:00 2023-01-24T17:27:29.237 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:27:29.238 INFO:teuthology.orchestra.run.smithi077.stdout:Total 169 kB/s | 90 kB 00:00 2023-01-24T17:27:29.238 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:27:29.247 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:27:29.247 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:27:29.299 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:27:29.302 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:27:29.417 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:27:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:30.449 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:27:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:27:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:30.450 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:30.644 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-mirror 2023-01-24T17:27:31.366 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:43 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:32.508 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:32.509 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:32.509 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2023-01-24T17:27:32.510 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:32.510 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:27:32.510 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 3.5 M 2023-01-24T17:27:32.510 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:32.510 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:27:32.511 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:32.511 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-01-24T17:27:32.511 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:32.511 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 3.5 M 2023-01-24T17:27:32.511 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 16 M 2023-01-24T17:27:32.512 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:27:33.283 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-mirror-17.2.5-727.g49f8fb05.el8.x86_64.rpm 4.5 MB/s | 3.5 MB 00:00 2023-01-24T17:27:33.283 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:27:33.283 INFO:teuthology.orchestra.run.smithi077.stdout:Total 4.5 MB/s | 3.5 MB 00:00 2023-01-24T17:27:33.284 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:27:33.295 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:27:33.295 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:27:33.358 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:27:33.361 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:27:33.841 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:27:34.067 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:34.505 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:35.088 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:35.088 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:35.088 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:27:35.088 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:27:35.089 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:35.089 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:35.297 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-nbd 2023-01-24T17:27:36.045 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:48 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:27:37.189 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:27:37.190 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:37.190 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2023-01-24T17:27:37.190 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:37.191 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:27:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:17.2.5-727.g49f8fb05.el8 ceph 172 k 2023-01-24T17:27:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:37.191 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:27:37.191 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:27:37.192 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-01-24T17:27:37.192 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:37.192 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 172 k 2023-01-24T17:27:37.193 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 548 k 2023-01-24T17:27:37.193 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:27:37.386 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-nbd-17.2.5-727.g49f8fb05.el8.x86_64.rpm 887 kB/s | 172 kB 00:00 2023-01-24T17:27:37.386 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:27:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:Total 881 kB/s | 172 kB 00:00 2023-01-24T17:27:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:27:37.397 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:27:37.397 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:27:37.450 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:27:37.453 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:27:37.576 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:27:37.755 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:38.059 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-nbd-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:38.524 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:17.2.5-727.g49f8fb05.el8.x86_64 1/1 2023-01-24T17:27:38.524 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:38.525 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:27:38.525 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:17.2.5-727.g49f8fb05.el8.x86_64 2023-01-24T17:27:38.525 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:38.525 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:27:38.755 DEBUG:teuthology.parallel:result is None 2023-01-24T17:27:38.755 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2023-01-24T17:27:38.793 INFO:teuthology.orchestra.run.smithi077.stdout:CentOSStream 2023-01-24T17:27:38.794 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-24T17:27:38.795 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T17:27:39.021 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-24T17:27:39.043 INFO:teuthology.orchestra.run.smithi077.stdout:17.2.5-727.g49f8fb05.el8 2023-01-24T17:27:39.044 INFO:teuthology.packaging:The installed version of ceph is 17.2.5-727.g49f8fb05.el8 2023-01-24T17:27:39.044 INFO:teuthology.task.install:The correct ceph version 17.2.5-727.g49f8fb05 is installed. 2023-01-24T17:27:39.044 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-24T17:27:39.045 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:39.045 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-24T17:27:39.110 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-24T17:27:39.110 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:39.111 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2023-01-24T17:27:39.177 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-24T17:27:39.245 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-24T17:27:39.246 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:39.246 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-24T17:27:39.312 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-24T17:27:39.379 INFO:teuthology.run_tasks:Running task ceph... 2023-01-24T17:27:39.441 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-01-24T17:27:39.442 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /var/log/ceph 2023-01-24T17:27:39.469 INFO:tasks.ceph:Disabling ceph logrotate... 2023-01-24T17:27:39.470 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-24T17:27:39.537 INFO:tasks.ceph:Creating extra log directories... 2023-01-24T17:27:39.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-24T17:27:39.612 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-01-24T17:27:39.613 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/export/encryption-key/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-01-24T17:27:39.613 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135503', 'branch': 'wip-yuri7-testing-2023-01-23-1532-quincy', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}}', 'email': 'yweinstein@radhat.com', 'first_in_suite': False, 'job_id': '7135503', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri7-testing-2023-01-23-1532-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/export/encryption-key/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}}, 'rgw': {'client.0': {'use-vault-role': 'client.0'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri7-testing-2023-01-23-1532-quincy', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuri7-testing-2023-01-23-1532-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'targets': {'smithi077.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjvvfvCe9LCqROmpxYy9mNZms0xnbU9bEpnlulym6um/jVtB8UK2T0VosXV51jyFB6VLBov5o2EIMYIOTn95U0iWcC8Qhy4DbR2o4becRkORALtzBdfb4zZLO6lQq1wnn5s9eIFHcMETtf+93jGiObxCkrYrfMYEeGuBiHykJwLLTWgGZmxVAUISBc6GjG/HrjXvrczl9ahaWMVQ8KPxP5x0R+aRo/HHUH9DORY4GVcoCaQsh0DyCKJ/bPY2tFgsqR7XEiwvgTftzKLYboZPRO6mE82QIt2QXb3ynrxMGIdL6ZCSNqxoaxkJz1YVXS4JGHy8TLOnczYyI+iuHqR48NRJZdC/antLW+flG//2mbo56YV2zLE+icC7mn8HtBtQkO9OpNlKTFA8uulAmUftDVGMHmp0NprqCLxeja9pMnnQ+wFLV1mnhAGhjMgN6Q+P5T/BjVBKI0vfjBWWLxu+vXuOX/1V6iSICkNsrg5n6kT5VBuC1gJN4EY9Z8W98Juxs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'vault': {'client.0': {'engine': 'transit', 'flavor': 'old', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/transit/keys/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1'}, {'path': 'my-key-2'}]}}}, {'rgw': {'client.0': None}}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8e1bba12e584b7ae912e92e94afe9c075ab51fa3', 'timestamp': '2023-01-24_16:04:03', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.17735'} 2023-01-24T17:27:39.614 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-24T17:27:39.663 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2023-01-24T17:27:39.731 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:39.732 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2023-01-24T17:27:39.790 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-24T17:27:39.791 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2023-01-24T17:27:39.848 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-24T17:27:39.848 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T17:27:39.849 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 373879 Links: 1 2023-01-24T17:27:39.849 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T17:27:39.849 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T17:27:39.849 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-01-24 17:27:34.244072519 +0000 2023-01-24T17:27:39.849 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-01-24 17:23:58.672676067 +0000 2023-01-24T17:27:39.849 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-01-24 17:23:58.672676067 +0000 2023-01-24T17:27:39.850 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-01-24T17:27:39.850 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-24T17:27:39.914 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-01-24T17:27:39.914 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-01-24T17:27:39.914 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000238267 s, 2.1 MB/s 2023-01-24T17:27:39.915 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-24T17:27:39.974 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2023-01-24T17:27:40.032 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-24T17:27:40.032 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T17:27:40.033 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 371201 Links: 1 2023-01-24T17:27:40.033 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T17:27:40.033 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T17:27:40.033 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-01-24 17:27:34.244072519 +0000 2023-01-24T17:27:40.033 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-01-24 17:23:59.192664070 +0000 2023-01-24T17:27:40.033 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-01-24 17:23:59.192664070 +0000 2023-01-24T17:27:40.034 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-01-24T17:27:40.034 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-24T17:27:40.098 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-01-24T17:27:40.099 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-01-24T17:27:40.099 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000225146 s, 2.3 MB/s 2023-01-24T17:27:40.100 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-24T17:27:40.160 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2023-01-24T17:27:40.218 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-24T17:27:40.218 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T17:27:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 371350 Links: 1 2023-01-24T17:27:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T17:27:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T17:27:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-01-24 17:27:34.244072519 +0000 2023-01-24T17:27:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-01-24 17:23:59.732651610 +0000 2023-01-24T17:27:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-01-24 17:23:59.732651610 +0000 2023-01-24T17:27:40.220 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-01-24T17:27:40.220 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-24T17:27:40.284 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-01-24T17:27:40.284 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-01-24T17:27:40.285 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000214992 s, 2.4 MB/s 2023-01-24T17:27:40.286 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-24T17:27:40.345 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2023-01-24T17:27:40.404 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-24T17:27:40.404 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T17:27:40.405 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 371449 Links: 1 2023-01-24T17:27:40.405 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T17:27:40.405 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T17:27:40.405 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-01-24 17:27:34.244072519 +0000 2023-01-24T17:27:40.405 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-01-24 17:24:00.246639750 +0000 2023-01-24T17:27:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-01-24 17:24:00.246639750 +0000 2023-01-24T17:27:40.406 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-01-24T17:27:40.406 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-24T17:27:40.470 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-01-24T17:27:40.470 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-01-24T17:27:40.471 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000156709 s, 3.3 MB/s 2023-01-24T17:27:40.472 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-24T17:27:40.531 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T17:27:40.531 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-01-24T17:27:40.531 INFO:tasks.ceph:Generating config... 2023-01-24T17:27:40.533 INFO:tasks.ceph:[client] debug rgw = 20 2023-01-24T17:27:40.533 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-01-24T17:27:40.533 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = vault 2023-01-24T17:27:40.534 INFO:tasks.ceph:[client] rgw crypt vault auth = token 2023-01-24T17:27:40.534 INFO:tasks.ceph:[client] rgw crypt vault prefix = /v1/transit/export/encryption-key/ 2023-01-24T17:27:40.534 INFO:tasks.ceph:[client] rgw crypt vault secret engine = transit 2023-01-24T17:27:40.534 INFO:tasks.ceph:[client] setgroup = ceph 2023-01-24T17:27:40.535 INFO:tasks.ceph:[client] setuser = ceph 2023-01-24T17:27:40.535 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-01-24T17:27:40.535 INFO:tasks.ceph:[mgr] debug ms = 1 2023-01-24T17:27:40.535 INFO:tasks.ceph:[mon] debug mon = 20 2023-01-24T17:27:40.536 INFO:tasks.ceph:[mon] debug ms = 1 2023-01-24T17:27:40.536 INFO:tasks.ceph:[mon] debug paxos = 20 2023-01-24T17:27:40.536 INFO:tasks.ceph:[osd] debug ms = 1 2023-01-24T17:27:40.537 INFO:tasks.ceph:[osd] debug osd = 20 2023-01-24T17:27:40.537 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-01-24T17:27:40.537 INFO:tasks.ceph:Setting up mon.a... 2023-01-24T17:27:40.538 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-01-24T17:27:40.624 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.keyring 2023-01-24T17:27:40.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-01-24T17:27:40.668 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T17:27:40.735 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.77')] 2023-01-24T17:27:40.736 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.77'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/export/encryption-key/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2023-01-24T17:27:40.737 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:40.737 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T17:27:40.793 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.77 --print /home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:27:40.871 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:27:40.872 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:27:40.873 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:27:40.873 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: generated fsid 42f52ac0-10ca-44e3-9331-cf221ec3091f 2023-01-24T17:27:40.873 INFO:teuthology.orchestra.run.smithi077.stdout:setting min_mon_release = octopus 2023-01-24T17:27:40.874 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 0 2023-01-24T17:27:40.874 INFO:teuthology.orchestra.run.smithi077.stdout:fsid 42f52ac0-10ca-44e3-9331-cf221ec3091f 2023-01-24T17:27:40.874 INFO:teuthology.orchestra.run.smithi077.stdout:last_changed 2023-01-24T17:27:40.870945+0000 2023-01-24T17:27:40.874 INFO:teuthology.orchestra.run.smithi077.stdout:created 2023-01-24T17:27:40.870945+0000 2023-01-24T17:27:40.874 INFO:teuthology.orchestra.run.smithi077.stdout:min_mon_release 15 (octopus) 2023-01-24T17:27:40.874 INFO:teuthology.orchestra.run.smithi077.stdout:election_strategy: 1 2023-01-24T17:27:40.875 INFO:teuthology.orchestra.run.smithi077.stdout:0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2023-01-24T17:27:40.875 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-01-24T17:27:40.875 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T17:27:40.931 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 42f52ac0-10ca-44e3-9331-cf221ec3091f... 2023-01-24T17:27:40.932 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-24T17:27:41.016 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2023-01-24T17:27:41.016 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2023-01-24T17:27:41.017 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-24T17:27:41.017 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2023-01-24T17:27:41.017 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.017 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2023-01-24T17:27:41.018 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.018 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2023-01-24T17:27:41.018 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.018 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2023-01-24T17:27:41.018 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2023-01-24T17:27:41.019 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.019 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2023-01-24T17:27:41.019 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2023-01-24T17:27:41.019 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.019 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-24T17:27:41.019 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2023-01-24T17:27:41.020 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.020 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2023-01-24T17:27:41.020 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-24T17:27:41.020 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.020 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2023-01-24T17:27:41.021 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.021 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2023-01-24T17:27:41.021 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2023-01-24T17:27:41.021 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2023-01-24T17:27:41.021 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2023-01-24T17:27:41.021 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2023-01-24T17:27:41.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2023-01-24T17:27:41.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2023-01-24T17:27:41.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-24T17:27:41.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-24T17:27:41.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2023-01-24T17:27:41.023 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.023 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-24T17:27:41.023 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.023 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2023-01-24T17:27:41.023 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.023 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2023-01-24T17:27:41.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.024 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2023-01-24T17:27:41.024 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2023-01-24T17:27:41.024 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2023-01-24T17:27:41.024 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = 172.21.15.77 2023-01-24T17:27:41.025 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 42f52ac0-10ca-44e3-9331-cf221ec3091f 2023-01-24T17:27:41.025 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.025 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2023-01-24T17:27:41.025 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2023-01-24T17:27:41.025 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.025 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2023-01-24T17:27:41.026 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2023-01-24T17:27:41.026 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.026 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2023-01-24T17:27:41.026 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2023-01-24T17:27:41.026 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.027 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2023-01-24T17:27:41.027 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2023-01-24T17:27:41.027 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2023-01-24T17:27:41.027 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.027 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2023-01-24T17:27:41.027 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2023-01-24T17:27:41.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.028 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2023-01-24T17:27:41.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.028 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2023-01-24T17:27:41.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.029 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2023-01-24T17:27:41.029 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.029 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2023-01-24T17:27:41.029 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2023-01-24T17:27:41.029 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.029 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2023-01-24T17:27:41.030 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2023-01-24T17:27:41.030 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-01-24T17:27:41.030 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2023-01-24T17:27:41.030 INFO:teuthology.orchestra.run.smithi077.stdout: osd shutdown pgref assert = True 2023-01-24T17:27:41.030 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.031 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2023-01-24T17:27:41.031 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-01-24T17:27:41.031 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2023-01-24T17:27:41.031 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2023-01-24T17:27:41.031 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2023-01-24T17:27:41.031 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2023-01-24T17:27:41.032 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2023-01-24T17:27:41.032 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2023-01-24T17:27:41.032 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.032 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2023-01-24T17:27:41.032 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-01-24T17:27:41.033 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2023-01-24T17:27:41.033 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2023-01-24T17:27:41.033 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2023-01-24T17:27:41.033 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2023-01-24T17:27:41.033 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2023-01-24T17:27:41.034 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2023-01-24T17:27:41.034 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2023-01-24T17:27:41.034 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2023-01-24T17:27:41.034 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2023-01-24T17:27:41.034 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.034 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-24T17:27:41.035 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2023-01-24T17:27:41.035 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2023-01-24T17:27:41.035 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.035 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2023-01-24T17:27:41.035 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-24T17:27:41.036 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-24T17:27:41.036 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.036 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2023-01-24T17:27:41.036 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2023-01-24T17:27:41.036 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.036 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2023-01-24T17:27:41.037 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:41.037 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2023-01-24T17:27:41.037 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2023-01-24T17:27:41.037 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2023-01-24T17:27:41.037 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2023-01-24T17:27:41.038 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-24T17:27:41.038 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-24T17:27:41.038 INFO:teuthology.orchestra.run.smithi077.stdout: debug rgw = 20 2023-01-24T17:27:41.038 INFO:teuthology.orchestra.run.smithi077.stdout: rgw crypt require ssl = False 2023-01-24T17:27:41.038 INFO:teuthology.orchestra.run.smithi077.stdout: rgw crypt s3 kms backend = vault 2023-01-24T17:27:41.038 INFO:teuthology.orchestra.run.smithi077.stdout: rgw crypt vault auth = token 2023-01-24T17:27:41.039 INFO:teuthology.orchestra.run.smithi077.stdout: rgw crypt vault prefix = /v1/transit/export/encryption-key/ 2023-01-24T17:27:41.039 INFO:teuthology.orchestra.run.smithi077.stdout: rgw crypt vault secret engine = transit 2023-01-24T17:27:41.039 INFO:teuthology.orchestra.run.smithi077.stdout: setgroup = ceph 2023-01-24T17:27:41.039 INFO:teuthology.orchestra.run.smithi077.stdout: setuser = ceph 2023-01-24T17:27:41.039 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2023-01-24T17:27:41.041 INFO:tasks.ceph:Creating admin key on mon.a... 2023-01-24T17:27:41.041 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-01-24T17:27:41.135 INFO:tasks.ceph:Copying monmap to all nodes... 2023-01-24T17:27:41.135 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:41.136 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-01-24T17:27:41.153 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:41.154 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-01-24T17:27:41.211 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2023-01-24T17:27:41.211 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:41.212 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-24T17:27:41.212 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T17:27:41.288 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:41.289 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:27:41.346 INFO:tasks.ceph:Setting up mon nodes... 2023-01-24T17:27:41.347 INFO:tasks.ceph:Setting up mgr nodes... 2023-01-24T17:27:41.347 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T17:27:41.436 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T17:27:41.438 INFO:tasks.ceph:Setting up mds nodes... 2023-01-24T17:27:41.438 INFO:tasks.ceph_client:Setting up client nodes... 2023-01-24T17:27:41.439 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-01-24T17:27:41.479 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-01-24T17:27:41.493 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-01-24T17:27:41.493 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-01-24T17:27:41.494 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-01-24T17:27:41.561 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T17:27:41.561 INFO:tasks.ceph:role: osd.0 2023-01-24T17:27:41.561 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com 2023-01-24T17:27:41.562 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-24T17:27:41.635 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T17:27:41.635 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T17:27:41.636 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T17:27:41.636 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T17:27:41.636 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T17:27:41.636 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2023-01-24T17:27:41.637 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T17:27:41.637 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T17:27:41.637 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T17:27:41.637 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T17:27:42.058 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2023-01-24T17:27:42.059 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2023-01-24T17:27:42.060 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-01-24T17:27:42.205 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-01-24T17:27:42.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-01-24T17:27:42.305 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T17:27:42.305 INFO:tasks.ceph:role: osd.1 2023-01-24T17:27:42.306 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com 2023-01-24T17:27:42.306 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-24T17:27:42.380 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T17:27:42.380 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T17:27:42.380 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T17:27:42.381 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T17:27:42.381 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T17:27:42.381 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2023-01-24T17:27:42.381 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T17:27:42.381 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T17:27:42.381 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T17:27:42.382 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T17:27:42.809 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2023-01-24T17:27:42.810 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2023-01-24T17:27:42.811 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-01-24T17:27:42.923 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-01-24T17:27:42.955 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-01-24T17:27:43.023 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T17:27:43.023 INFO:tasks.ceph:role: osd.2 2023-01-24T17:27:43.023 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com 2023-01-24T17:27:43.024 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-01-24T17:27:43.099 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T17:27:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T17:27:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T17:27:43.100 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-24T17:27:43.100 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T17:27:43.100 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2023-01-24T17:27:43.100 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T17:27:43.100 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T17:27:43.101 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T17:27:43.101 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T17:27:43.544 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2023-01-24T17:27:43.548 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2023-01-24T17:27:43.548 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-01-24T17:27:43.704 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-01-24T17:27:43.739 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:27:43.845 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:43.842+0000 7f6f55a35540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-01-24T17:27:43.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:43.845+0000 7f6f55a35540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-01-24T17:27:43.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:43.845+0000 7f6f55a35540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T17:27:43.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:43.845+0000 7f6f55a35540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T17:27:43.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:43.845+0000 7f6f55a35540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T17:27:43.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:43.846+0000 7f6f55a35540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-01-24T17:27:45.921 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-01-24T17:27:45.951 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:27:46.037 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:46.033+0000 7fe0c65ab540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-01-24T17:27:46.037 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:46.034+0000 7fe0c65ab540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-01-24T17:27:46.037 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:46.034+0000 7fe0c65ab540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T17:27:46.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:46.034+0000 7fe0c65ab540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T17:27:46.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:46.034+0000 7fe0c65ab540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T17:27:46.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:46.034+0000 7fe0c65ab540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-01-24T17:27:48.112 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-01-24T17:27:48.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:27:48.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:48.226+0000 7f6828f4b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-01-24T17:27:48.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:48.226+0000 7f6828f4b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-01-24T17:27:48.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:48.226+0000 7f6828f4b540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T17:27:48.230 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:48.226+0000 7f6828f4b540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T17:27:48.230 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:48.226+0000 7f6828f4b540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T17:27:48.230 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:27:48.227+0000 7f6828f4b540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-01-24T17:27:50.294 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-01-24T17:27:50.363 INFO:tasks.ceph:Reading keys from all nodes... 2023-01-24T17:27:50.364 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:50.364 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-01-24T17:27:50.432 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:50.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-01-24T17:27:50.502 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:50.502 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-01-24T17:27:50.570 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:50.570 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-01-24T17:27:50.637 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:50.638 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-01-24T17:27:50.695 INFO:tasks.ceph:Adding keys to all mons... 2023-01-24T17:27:50.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-24T17:27:50.761 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.x] 2023-01-24T17:27:50.761 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQANFdBjrJLuGRAAXeXUlNNujGosXR2Df1IYvA== 2023-01-24T17:27:50.761 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2023-01-24T17:27:50.761 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAPFdBjdXNNMhAACSmw6IId2lVWJdZ7QR8NqQ== 2023-01-24T17:27:50.762 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2023-01-24T17:27:50.762 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQASFdBjf4MWAhAA49vM6//vYdK3R93o4EA/yg== 2023-01-24T17:27:50.762 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2023-01-24T17:27:50.762 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAUFdBjFV2PDRAAjAVuZsrBLDiKB80SgAgwiQ== 2023-01-24T17:27:50.762 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2023-01-24T17:27:50.762 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQANFdBjYiqCHBAAQsTCi+agq0jjSY4eJ4Wb4Q== 2023-01-24T17:27:50.763 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-24T17:27:50.853 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T17:27:50.903 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T17:27:50.952 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T17:27:51.003 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-24T17:27:51.056 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-01-24T17:27:51.056 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-01-24T17:27:51.083 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-24T17:27:51.177 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-01-24T17:27:51.205 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-24T17:27:51.263 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-01-24T17:27:51.263 INFO:tasks.ceph.mon.a:Restarting daemon 2023-01-24T17:27:51.264 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-01-24T17:27:51.305 INFO:tasks.ceph.mon.a:Started 2023-01-24T17:27:51.306 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-01-24T17:27:51.306 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-01-24T17:27:51.306 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-01-24T17:27:51.308 INFO:tasks.ceph.mgr.x:Started 2023-01-24T17:27:51.309 INFO:tasks.ceph:Setting crush tunables to default 2023-01-24T17:27:51.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd crush tunables default 2023-01-24T17:27:51.511 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2023-01-24T17:27:51.521 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-01-24T17:27:51.522 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:51.522 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-01-24T17:27:51.549 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:51.549 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-01-24T17:27:51.617 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:27:51.617 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-01-24T17:27:51.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd new 9e67d55f-2f53-4427-9576-e00932ec8066 0 2023-01-24T17:27:51.949 INFO:teuthology.orchestra.run.smithi077.stdout:0 2023-01-24T17:27:51.962 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd new 292dd058-5168-46f0-8c25-8e5981e647a4 1 2023-01-24T17:27:52.170 INFO:teuthology.orchestra.run.smithi077.stdout:1 2023-01-24T17:27:52.182 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd new 3abf1aa7-dfb2-452e-b61c-f3b192ca00c2 2 2023-01-24T17:27:52.421 INFO:teuthology.orchestra.run.smithi077.stdout:2 2023-01-24T17:27:52.436 INFO:tasks.ceph.osd.0:Restarting daemon 2023-01-24T17:27:52.438 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-01-24T17:27:52.442 INFO:tasks.ceph.osd.0:Started 2023-01-24T17:27:52.442 INFO:tasks.ceph.osd.1:Restarting daemon 2023-01-24T17:27:52.443 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-01-24T17:27:52.445 INFO:tasks.ceph.osd.1:Started 2023-01-24T17:27:52.445 INFO:tasks.ceph.osd.2:Restarting daemon 2023-01-24T17:27:52.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-01-24T17:27:52.448 INFO:tasks.ceph.osd.2:Started 2023-01-24T17:27:52.448 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T17:27:52.562 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:52.559+0000 7fb2bdc46180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-24T17:27:52.634 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:52.632+0000 7fb2bdc46180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-24T17:27:52.661 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:27:52.661 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":5,"fsid":"42f52ac0-10ca-44e3-9331-cf221ec3091f","created":"2023-01-24T17:27:51.417263+0000","modified":"2023-01-24T17:27:52.418578+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T17:27:52.418578+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"9e67d55f-2f53-4427-9576-e00932ec8066","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"292dd058-5168-46f0-8c25-8e5981e647a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3abf1aa7-dfb2-452e-b61c-f3b192ca00c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:27:52.673 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-01-24T17:27:52.673 INFO:tasks.ceph:Waiting for OSDs to come up 2023-01-24T17:27:52.715 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:52.712+0000 7fb2bdc46180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-24T17:27:52.797 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-01-24T17:27:52.794+0000 7f40ff83d540 -1 Falling back to public interface 2023-01-24T17:27:52.897 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:52.895+0000 7fb2bdc46180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-24T17:27:52.975 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T17:27:53.048 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-01-24T17:27:53.045+0000 7fc8cd0d6540 -1 Falling back to public interface 2023-01-24T17:27:53.061 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-01-24T17:27:53.058+0000 7f8ca4e41540 -1 Falling back to public interface 2023-01-24T17:27:53.163 INFO:teuthology.misc.health.smithi077.stdout: 2023-01-24T17:27:53.163 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":5,"fsid":"42f52ac0-10ca-44e3-9331-cf221ec3091f","created":"2023-01-24T17:27:51.417263+0000","modified":"2023-01-24T17:27:52.418578+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T17:27:52.418578+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"9e67d55f-2f53-4427-9576-e00932ec8066","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"292dd058-5168-46f0-8c25-8e5981e647a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3abf1aa7-dfb2-452e-b61c-f3b192ca00c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:27:53.174 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-01-24T17:27:53.674 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:53.671+0000 7fb2bdc46180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-01-24T17:27:53.812 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:53.809+0000 7fb2bdc46180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-24T17:27:54.030 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:54.027+0000 7fb2bdc46180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-24T17:27:54.099 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:54.096+0000 7fb2bdc46180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-24T17:27:54.234 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:54.231+0000 7fb2bdc46180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-24T17:27:54.482 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:54.480+0000 7fb2bdc46180 -1 mgr[py] Module not found: 'prometheus' 2023-01-24T17:27:54.483 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:54.480+0000 7fb2bdc46180 -1 mgr[py] Traceback (most recent call last): 2023-01-24T17:27:54.483 INFO:tasks.ceph.mgr.x.smithi077.stderr: File "/usr/share/ceph/mgr/prometheus/__init__.py", line 2, in 2023-01-24T17:27:54.483 INFO:tasks.ceph.mgr.x.smithi077.stderr: from .module import Module, StandbyModule 2023-01-24T17:27:54.484 INFO:tasks.ceph.mgr.x.smithi077.stderr: File "/usr/share/ceph/mgr/prometheus/module.py", line 3, in 2023-01-24T17:27:54.484 INFO:tasks.ceph.mgr.x.smithi077.stderr: from packaging.version import Version 2023-01-24T17:27:54.484 INFO:tasks.ceph.mgr.x.smithi077.stderr:ModuleNotFoundError: No module named 'packaging' 2023-01-24T17:27:54.485 INFO:tasks.ceph.mgr.x.smithi077.stderr: 2023-01-24T17:27:54.485 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:54.480+0000 7fb2bdc46180 -1 mgr[py] Class not found in module 'prometheus' 2023-01-24T17:27:54.485 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:54.480+0000 7fb2bdc46180 -1 mgr[py] Error loading module 'prometheus': (2) No such file or directory 2023-01-24T17:27:54.661 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:54.659+0000 7fb2bdc46180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-24T17:27:54.799 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:54.797+0000 7fb2bdc46180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-24T17:27:54.912 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:54.909+0000 7fb2bdc46180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-24T17:27:55.051 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:55.049+0000 7fb2bdc46180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-24T17:27:55.128 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:55.125+0000 7fb2bdc46180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-24T17:27:55.344 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:55.341+0000 7fb2bdc46180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-24T17:27:55.420 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:55.417+0000 7fb2bdc46180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-24T17:27:55.500 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-01-24T17:27:55.497+0000 7f40ff83d540 -1 osd.0 0 log_to_monitors true 2023-01-24T17:27:55.785 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-01-24T17:27:55.783+0000 7fc8cd0d6540 -1 osd.1 0 log_to_monitors true 2023-01-24T17:27:55.799 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-01-24T17:27:55.797+0000 7f8ca4e41540 -1 osd.2 0 log_to_monitors true 2023-01-24T17:27:56.044 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:56.041+0000 7fb2bdc46180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-24T17:27:56.124 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:56.121+0000 7fb2bdc46180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-24T17:27:56.368 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:56.365+0000 7fb2bdc46180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-24T17:27:56.449 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:56.446+0000 7fb2bdc46180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-24T17:27:56.650 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:56.648+0000 7fb2bdc46180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-24T17:27:56.725 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:56.723+0000 7fb2bdc46180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-24T17:27:56.886 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:56.884+0000 7fb2bdc46180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-24T17:27:56.887 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:56.884+0000 7fb2bdc46180 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: prometheus 2023-01-24T17:27:57.044 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-01-24T17:27:57.042+0000 7f40f6448700 -1 osd.0 0 waiting for initial osdmap 2023-01-24T17:27:57.046 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-01-24T17:27:57.043+0000 7f40ef23e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T17:27:57.971 INFO:tasks.ceph.mgr.x.smithi077.stderr:2023-01-24T17:27:57.969+0000 7fb27a671700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-01-24T17:27:58.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-01-24T17:27:58.412+0000 7fc8c3ce1700 -1 osd.1 0 waiting for initial osdmap 2023-01-24T17:27:58.417 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-01-24T17:27:58.414+0000 7fc8bcad7700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T17:27:58.430 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-01-24T17:27:58.427+0000 7f8c9ba4c700 -1 osd.2 0 waiting for initial osdmap 2023-01-24T17:27:58.433 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-01-24T17:27:58.431+0000 7f8c94842700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T17:27:59.477 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T17:27:59.784 INFO:teuthology.misc.health.smithi077.stdout: 2023-01-24T17:27:59.785 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":9,"fsid":"42f52ac0-10ca-44e3-9331-cf221ec3091f","created":"2023-01-24T17:27:51.417263+0000","modified":"2023-01-24T17:27:59.413896+0000","last_up_change":"2023-01-24T17:27:59.413896+0000","last_in_change":"2023-01-24T17:27:52.418578+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"9e67d55f-2f53-4427-9576-e00932ec8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":103662},{"type":"v1","addr":"172.21.15.77:6801","nonce":103662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":103662},{"type":"v1","addr":"172.21.15.77:6803","nonce":103662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":103662},{"type":"v1","addr":"172.21.15.77:6807","nonce":103662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":103662},{"type":"v1","addr":"172.21.15.77:6805","nonce":103662}]},"public_addr":"172.21.15.77:6801/103662","cluster_addr":"172.21.15.77:6803/103662","heartbeat_back_addr":"172.21.15.77:6807/103662","heartbeat_front_addr":"172.21.15.77:6805/103662","state":["exists","up"]},{"osd":1,"uuid":"292dd058-5168-46f0-8c25-8e5981e647a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":103663},{"type":"v1","addr":"172.21.15.77:6809","nonce":103663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":103663},{"type":"v1","addr":"172.21.15.77:6811","nonce":103663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":103663},{"type":"v1","addr":"172.21.15.77:6815","nonce":103663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":103663},{"type":"v1","addr":"172.21.15.77:6813","nonce":103663}]},"public_addr":"172.21.15.77:6809/103663","cluster_addr":"172.21.15.77:6811/103663","heartbeat_back_addr":"172.21.15.77:6815/103663","heartbeat_front_addr":"172.21.15.77:6813/103663","state":["exists","up"]},{"osd":2,"uuid":"3abf1aa7-dfb2-452e-b61c-f3b192ca00c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":103693},{"type":"v1","addr":"172.21.15.77:6817","nonce":103693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":103693},{"type":"v1","addr":"172.21.15.77:6819","nonce":103693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":103693},{"type":"v1","addr":"172.21.15.77:6823","nonce":103693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":103693},{"type":"v1","addr":"172.21.15.77:6821","nonce":103693}]},"public_addr":"172.21.15.77:6817/103693","cluster_addr":"172.21.15.77:6819/103693","heartbeat_back_addr":"172.21.15.77:6823/103693","heartbeat_front_addr":"172.21.15.77:6821/103693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:27:56.533062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:27:59.797 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-01-24T17:27:59.797 INFO:tasks.ceph:Creating RBD pool 2023-01-24T17:27:59.797 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd pool create rbd 8 2023-01-24T17:28:00.574 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'rbd' created 2023-01-24T17:28:00.586 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-01-24T17:28:01.576 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rbd' on pool 'rbd' 2023-01-24T17:28:01.590 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-01-24T17:28:01.591 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-01-24T17:28:01.591 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-01-24T17:28:01.591 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-01-24T17:28:01.592 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-01-24T17:28:01.943 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:01.955 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":103255},{"type":"v1","addr":"172.21.15.77:6825","nonce":103255}]},"active_addr":"172.21.15.77:6825/103255","active_change":"2023-01-24T17:27:56.897869+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":false,"error_string":"No module named 'packaging'","module_options":{}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2138098189}]},{"addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":4130691959}]},{"addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":3835103044}]}]}} 2023-01-24T17:28:01.957 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-01-24T17:28:01.958 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-01-24T17:28:01.958 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T17:28:02.262 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:02.263 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":11,"fsid":"42f52ac0-10ca-44e3-9331-cf221ec3091f","created":"2023-01-24T17:27:51.417263+0000","modified":"2023-01-24T17:28:01.568048+0000","last_up_change":"2023-01-24T17:27:59.413896+0000","last_in_change":"2023-01-24T17:27:52.418578+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T17:28:00.113996+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T17:28:00.952800+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"9e67d55f-2f53-4427-9576-e00932ec8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":103662},{"type":"v1","addr":"172.21.15.77:6801","nonce":103662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":103662},{"type":"v1","addr":"172.21.15.77:6803","nonce":103662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":103662},{"type":"v1","addr":"172.21.15.77:6807","nonce":103662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":103662},{"type":"v1","addr":"172.21.15.77:6805","nonce":103662}]},"public_addr":"172.21.15.77:6801/103662","cluster_addr":"172.21.15.77:6803/103662","heartbeat_back_addr":"172.21.15.77:6807/103662","heartbeat_front_addr":"172.21.15.77:6805/103662","state":["exists","up"]},{"osd":1,"uuid":"292dd058-5168-46f0-8c25-8e5981e647a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":103663},{"type":"v1","addr":"172.21.15.77:6809","nonce":103663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":103663},{"type":"v1","addr":"172.21.15.77:6811","nonce":103663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":103663},{"type":"v1","addr":"172.21.15.77:6815","nonce":103663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":103663},{"type":"v1","addr":"172.21.15.77:6813","nonce":103663}]},"public_addr":"172.21.15.77:6809/103663","cluster_addr":"172.21.15.77:6811/103663","heartbeat_back_addr":"172.21.15.77:6815/103663","heartbeat_front_addr":"172.21.15.77:6813/103663","state":["exists","up"]},{"osd":2,"uuid":"3abf1aa7-dfb2-452e-b61c-f3b192ca00c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":103693},{"type":"v1","addr":"172.21.15.77:6817","nonce":103693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":103693},{"type":"v1","addr":"172.21.15.77:6819","nonce":103693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":103693},{"type":"v1","addr":"172.21.15.77:6823","nonce":103693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":103693},{"type":"v1","addr":"172.21.15.77:6821","nonce":103693}]},"public_addr":"172.21.15.77:6817/103693","cluster_addr":"172.21.15.77:6819/103693","heartbeat_back_addr":"172.21.15.77:6823/103693","heartbeat_front_addr":"172.21.15.77:6821/103693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:27:56.533062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:27:56.834326+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:27:56.759525+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:28:02.276 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-01-24T17:28:02.276 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T17:28:02.581 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:02.581 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":12,"fsid":"42f52ac0-10ca-44e3-9331-cf221ec3091f","created":"2023-01-24T17:27:51.417263+0000","modified":"2023-01-24T17:28:02.569860+0000","last_up_change":"2023-01-24T17:27:59.413896+0000","last_in_change":"2023-01-24T17:27:52.418578+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T17:28:00.113996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T17:28:00.952800+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9e67d55f-2f53-4427-9576-e00932ec8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":103662},{"type":"v1","addr":"172.21.15.77:6801","nonce":103662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":103662},{"type":"v1","addr":"172.21.15.77:6803","nonce":103662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":103662},{"type":"v1","addr":"172.21.15.77:6807","nonce":103662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":103662},{"type":"v1","addr":"172.21.15.77:6805","nonce":103662}]},"public_addr":"172.21.15.77:6801/103662","cluster_addr":"172.21.15.77:6803/103662","heartbeat_back_addr":"172.21.15.77:6807/103662","heartbeat_front_addr":"172.21.15.77:6805/103662","state":["exists","up"]},{"osd":1,"uuid":"292dd058-5168-46f0-8c25-8e5981e647a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":103663},{"type":"v1","addr":"172.21.15.77:6809","nonce":103663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":103663},{"type":"v1","addr":"172.21.15.77:6811","nonce":103663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":103663},{"type":"v1","addr":"172.21.15.77:6815","nonce":103663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":103663},{"type":"v1","addr":"172.21.15.77:6813","nonce":103663}]},"public_addr":"172.21.15.77:6809/103663","cluster_addr":"172.21.15.77:6811/103663","heartbeat_back_addr":"172.21.15.77:6815/103663","heartbeat_front_addr":"172.21.15.77:6813/103663","state":["exists","up"]},{"osd":2,"uuid":"3abf1aa7-dfb2-452e-b61c-f3b192ca00c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":103693},{"type":"v1","addr":"172.21.15.77:6817","nonce":103693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":103693},{"type":"v1","addr":"172.21.15.77:6819","nonce":103693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":103693},{"type":"v1","addr":"172.21.15.77:6823","nonce":103693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":103693},{"type":"v1","addr":"172.21.15.77:6821","nonce":103693}]},"public_addr":"172.21.15.77:6817/103693","cluster_addr":"172.21.15.77:6819/103693","heartbeat_back_addr":"172.21.15.77:6823/103693","heartbeat_front_addr":"172.21.15.77:6821/103693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:27:56.533062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:27:56.834326+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:27:56.759525+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:28:02.595 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-01-24T17:28:02.596 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-01-24T17:28:02.596 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-01-24T17:28:02.768 INFO:teuthology.orchestra.run.smithi077.stdout:38654705667 2023-01-24T17:28:02.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T17:28:02.774 INFO:teuthology.orchestra.run.smithi077.stdout:38654705667 2023-01-24T17:28:02.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T17:28:02.781 INFO:teuthology.orchestra.run.smithi077.stdout:34359738370 2023-01-24T17:28:02.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T17:28:03.147 INFO:teuthology.orchestra.run.smithi077.stdout:38654705666 2023-01-24T17:28:03.164 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.1 2023-01-24T17:28:03.173 INFO:teuthology.orchestra.run.smithi077.stdout:34359738369 2023-01-24T17:28:03.184 INFO:teuthology.orchestra.run.smithi077.stdout:38654705666 2023-01-24T17:28:03.186 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2023-01-24T17:28:03.197 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.2 2023-01-24T17:28:04.165 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T17:28:04.187 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T17:28:04.198 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T17:28:04.501 INFO:teuthology.orchestra.run.smithi077.stdout:38654705667 2023-01-24T17:28:04.515 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.1 2023-01-24T17:28:04.515 DEBUG:teuthology.parallel:result is None 2023-01-24T17:28:04.539 INFO:teuthology.orchestra.run.smithi077.stdout:38654705667 2023-01-24T17:28:04.541 INFO:teuthology.orchestra.run.smithi077.stdout:34359738370 2023-01-24T17:28:04.554 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.2 2023-01-24T17:28:04.554 DEBUG:teuthology.parallel:result is None 2023-01-24T17:28:04.555 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.0 2023-01-24T17:28:04.556 DEBUG:teuthology.parallel:result is None 2023-01-24T17:28:04.556 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-24T17:28:04.556 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T17:28:04.893 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:04.895 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-01-24T17:28:04.906 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-01-24T17:28:03.574943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328208,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":64,"num_write_kb":587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37,"ondisk_log_size":37,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":22968,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":22656,"kb_avail":314549832,"statfs":{"total":322122547200,"available":322099027968,"internally_reserved":0,"allocated":294912,"data_stored":74548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23199744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.661486"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.585236+0000","last_change":"2023-01-24T17:28:01.580089+0000","last_active":"2023-01-24T17:28:01.585236+0000","last_peered":"2023-01-24T17:28:01.585236+0000","last_clean":"2023-01-24T17:28:01.585236+0000","last_became_active":"2023-01-24T17:28:01.579822+0000","last_became_peered":"2023-01-24T17:28:01.579822+0000","last_unstale":"2023-01-24T17:28:01.585236+0000","last_undegraded":"2023-01-24T17:28:01.585236+0000","last_fullsized":"2023-01-24T17:28:01.585236+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T20:15:45.882651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.584887+0000","last_change":"2023-01-24T17:28:01.577395+0000","last_active":"2023-01-24T17:28:01.584887+0000","last_peered":"2023-01-24T17:28:01.584887+0000","last_clean":"2023-01-24T17:28:01.584887+0000","last_became_active":"2023-01-24T17:28:01.577154+0000","last_became_peered":"2023-01-24T17:28:01.577154+0000","last_unstale":"2023-01-24T17:28:01.584887+0000","last_undegraded":"2023-01-24T17:28:01.584887+0000","last_fullsized":"2023-01-24T17:28:01.584887+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T03:15:57.783329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.587888+0000","last_change":"2023-01-24T17:28:01.585043+0000","last_active":"2023-01-24T17:28:01.587888+0000","last_peered":"2023-01-24T17:28:01.587888+0000","last_clean":"2023-01-24T17:28:01.587888+0000","last_became_active":"2023-01-24T17:28:01.584794+0000","last_became_peered":"2023-01-24T17:28:01.584794+0000","last_unstale":"2023-01-24T17:28:01.587888+0000","last_undegraded":"2023-01-24T17:28:01.587888+0000","last_fullsized":"2023-01-24T17:28:01.587888+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T01:54:56.814018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.584919+0000","last_change":"2023-01-24T17:28:01.577392+0000","last_active":"2023-01-24T17:28:01.584919+0000","last_peered":"2023-01-24T17:28:01.584919+0000","last_clean":"2023-01-24T17:28:01.584919+0000","last_became_active":"2023-01-24T17:28:01.577131+0000","last_became_peered":"2023-01-24T17:28:01.577131+0000","last_unstale":"2023-01-24T17:28:01.584919+0000","last_undegraded":"2023-01-24T17:28:01.584919+0000","last_fullsized":"2023-01-24T17:28:01.584919+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T23:05:28.299805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.585092+0000","last_change":"2023-01-24T17:28:01.580274+0000","last_active":"2023-01-24T17:28:01.585092+0000","last_peered":"2023-01-24T17:28:01.585092+0000","last_clean":"2023-01-24T17:28:01.585092+0000","last_became_active":"2023-01-24T17:28:01.579970+0000","last_became_peered":"2023-01-24T17:28:01.579970+0000","last_unstale":"2023-01-24T17:28:01.585092+0000","last_undegraded":"2023-01-24T17:28:01.585092+0000","last_fullsized":"2023-01-24T17:28:01.585092+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T04:49:36.081845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"12'37","reported_seq":58,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T17:28:02.754254+0000","last_change":"2023-01-24T17:28:02.702907+0000","last_active":"2023-01-24T17:28:02.754254+0000","last_peered":"2023-01-24T17:28:02.754254+0000","last_clean":"2023-01-24T17:28:02.754254+0000","last_became_active":"2023-01-24T17:28:02.702195+0000","last_became_peered":"2023-01-24T17:28:02.702195+0000","last_unstale":"2023-01-24T17:28:02.754254+0000","last_undegraded":"2023-01-24T17:28:02.754254+0000","last_fullsized":"2023-01-24T17:28:02.754254+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:01.568048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:01.568048+0000","last_clean_scrub_stamp":"2023-01-24T17:28:01.568048+0000","objects_scrubbed":0,"log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T22:29:49.795887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328208,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":64,"num_write_kb":587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.590624+0000","last_change":"2023-01-24T17:28:01.585235+0000","last_active":"2023-01-24T17:28:01.590624+0000","last_peered":"2023-01-24T17:28:01.590624+0000","last_clean":"2023-01-24T17:28:01.590624+0000","last_became_active":"2023-01-24T17:28:01.585000+0000","last_became_peered":"2023-01-24T17:28:01.585000+0000","last_unstale":"2023-01-24T17:28:01.590624+0000","last_undegraded":"2023-01-24T17:28:01.590624+0000","last_fullsized":"2023-01-24T17:28:01.590624+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T00:45:00.286567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.585224+0000","last_change":"2023-01-24T17:28:01.580057+0000","last_active":"2023-01-24T17:28:01.585224+0000","last_peered":"2023-01-24T17:28:01.585224+0000","last_clean":"2023-01-24T17:28:01.585224+0000","last_became_active":"2023-01-24T17:28:01.579791+0000","last_became_peered":"2023-01-24T17:28:01.579791+0000","last_unstale":"2023-01-24T17:28:01.585224+0000","last_undegraded":"2023-01-24T17:28:01.585224+0000","last_fullsized":"2023-01-24T17:28:01.585224+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T04:55:38.137134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.587862+0000","last_change":"2023-01-24T17:28:01.585097+0000","last_active":"2023-01-24T17:28:01.587862+0000","last_peered":"2023-01-24T17:28:01.587862+0000","last_clean":"2023-01-24T17:28:01.587862+0000","last_became_active":"2023-01-24T17:28:01.584837+0000","last_became_peered":"2023-01-24T17:28:01.584837+0000","last_unstale":"2023-01-24T17:28:01.587862+0000","last_undegraded":"2023-01-24T17:28:01.587862+0000","last_fullsized":"2023-01-24T17:28:01.587862+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T22:23:05.991847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":328208,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":64,"num_write_kb":587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":590880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37,"ondisk_log_size":37,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7660,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849940,"statfs":{"total":107374182400,"available":107366338560,"internally_reserved":0,"allocated":102400,"data_stored":27186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7656,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849944,"statfs":{"total":107374182400,"available":107366342656,"internally_reserved":0,"allocated":98304,"data_stored":24020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]}]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7652,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849948,"statfs":{"total":107374182400,"available":107366346752,"internally_reserved":0,"allocated":94208,"data_stored":23342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T17:28:04.907 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T17:28:05.206 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:05.208 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-01-24T17:28:05.219 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-01-24T17:28:04.913779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328208,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":64,"num_write_kb":587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37,"ondisk_log_size":37,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":22968,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":22656,"kb_avail":314549832,"statfs":{"total":322122547200,"available":322099027968,"internally_reserved":0,"allocated":294912,"data_stored":74548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23199744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000322"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.585236+0000","last_change":"2023-01-24T17:28:01.580089+0000","last_active":"2023-01-24T17:28:01.585236+0000","last_peered":"2023-01-24T17:28:01.585236+0000","last_clean":"2023-01-24T17:28:01.585236+0000","last_became_active":"2023-01-24T17:28:01.579822+0000","last_became_peered":"2023-01-24T17:28:01.579822+0000","last_unstale":"2023-01-24T17:28:01.585236+0000","last_undegraded":"2023-01-24T17:28:01.585236+0000","last_fullsized":"2023-01-24T17:28:01.585236+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T20:15:45.882651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.584887+0000","last_change":"2023-01-24T17:28:01.577395+0000","last_active":"2023-01-24T17:28:01.584887+0000","last_peered":"2023-01-24T17:28:01.584887+0000","last_clean":"2023-01-24T17:28:01.584887+0000","last_became_active":"2023-01-24T17:28:01.577154+0000","last_became_peered":"2023-01-24T17:28:01.577154+0000","last_unstale":"2023-01-24T17:28:01.584887+0000","last_undegraded":"2023-01-24T17:28:01.584887+0000","last_fullsized":"2023-01-24T17:28:01.584887+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T03:15:57.783329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.587888+0000","last_change":"2023-01-24T17:28:01.585043+0000","last_active":"2023-01-24T17:28:01.587888+0000","last_peered":"2023-01-24T17:28:01.587888+0000","last_clean":"2023-01-24T17:28:01.587888+0000","last_became_active":"2023-01-24T17:28:01.584794+0000","last_became_peered":"2023-01-24T17:28:01.584794+0000","last_unstale":"2023-01-24T17:28:01.587888+0000","last_undegraded":"2023-01-24T17:28:01.587888+0000","last_fullsized":"2023-01-24T17:28:01.587888+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T01:54:56.814018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.584919+0000","last_change":"2023-01-24T17:28:01.577392+0000","last_active":"2023-01-24T17:28:01.584919+0000","last_peered":"2023-01-24T17:28:01.584919+0000","last_clean":"2023-01-24T17:28:01.584919+0000","last_became_active":"2023-01-24T17:28:01.577131+0000","last_became_peered":"2023-01-24T17:28:01.577131+0000","last_unstale":"2023-01-24T17:28:01.584919+0000","last_undegraded":"2023-01-24T17:28:01.584919+0000","last_fullsized":"2023-01-24T17:28:01.584919+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T23:05:28.299805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.585092+0000","last_change":"2023-01-24T17:28:01.580274+0000","last_active":"2023-01-24T17:28:01.585092+0000","last_peered":"2023-01-24T17:28:01.585092+0000","last_clean":"2023-01-24T17:28:01.585092+0000","last_became_active":"2023-01-24T17:28:01.579970+0000","last_became_peered":"2023-01-24T17:28:01.579970+0000","last_unstale":"2023-01-24T17:28:01.585092+0000","last_undegraded":"2023-01-24T17:28:01.585092+0000","last_fullsized":"2023-01-24T17:28:01.585092+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T04:49:36.081845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"12'37","reported_seq":58,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-01-24T17:28:02.754254+0000","last_change":"2023-01-24T17:28:02.702907+0000","last_active":"2023-01-24T17:28:02.754254+0000","last_peered":"2023-01-24T17:28:02.754254+0000","last_clean":"2023-01-24T17:28:02.754254+0000","last_became_active":"2023-01-24T17:28:02.702195+0000","last_became_peered":"2023-01-24T17:28:02.702195+0000","last_unstale":"2023-01-24T17:28:02.754254+0000","last_undegraded":"2023-01-24T17:28:02.754254+0000","last_fullsized":"2023-01-24T17:28:02.754254+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:01.568048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:01.568048+0000","last_clean_scrub_stamp":"2023-01-24T17:28:01.568048+0000","objects_scrubbed":0,"log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T22:29:49.795887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328208,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":64,"num_write_kb":587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.590624+0000","last_change":"2023-01-24T17:28:01.585235+0000","last_active":"2023-01-24T17:28:01.590624+0000","last_peered":"2023-01-24T17:28:01.590624+0000","last_clean":"2023-01-24T17:28:01.590624+0000","last_became_active":"2023-01-24T17:28:01.585000+0000","last_became_peered":"2023-01-24T17:28:01.585000+0000","last_unstale":"2023-01-24T17:28:01.590624+0000","last_undegraded":"2023-01-24T17:28:01.590624+0000","last_fullsized":"2023-01-24T17:28:01.590624+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T00:45:00.286567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.585224+0000","last_change":"2023-01-24T17:28:01.580057+0000","last_active":"2023-01-24T17:28:01.585224+0000","last_peered":"2023-01-24T17:28:01.585224+0000","last_clean":"2023-01-24T17:28:01.585224+0000","last_became_active":"2023-01-24T17:28:01.579791+0000","last_became_peered":"2023-01-24T17:28:01.579791+0000","last_unstale":"2023-01-24T17:28:01.585224+0000","last_undegraded":"2023-01-24T17:28:01.585224+0000","last_fullsized":"2023-01-24T17:28:01.585224+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T04:55:38.137134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-01-24T17:28:01.587862+0000","last_change":"2023-01-24T17:28:01.585097+0000","last_active":"2023-01-24T17:28:01.587862+0000","last_peered":"2023-01-24T17:28:01.587862+0000","last_clean":"2023-01-24T17:28:01.587862+0000","last_became_active":"2023-01-24T17:28:01.584837+0000","last_became_peered":"2023-01-24T17:28:01.584837+0000","last_unstale":"2023-01-24T17:28:01.587862+0000","last_undegraded":"2023-01-24T17:28:01.587862+0000","last_fullsized":"2023-01-24T17:28:01.587862+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T17:28:00.562728+0000","last_clean_scrub_stamp":"2023-01-24T17:28:00.562728+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T22:23:05.991847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":328208,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":64,"num_write_kb":587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":590880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37,"ondisk_log_size":37,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7660,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849940,"statfs":{"total":107374182400,"available":107366338560,"internally_reserved":0,"allocated":102400,"data_stored":27186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7656,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849944,"statfs":{"total":107374182400,"available":107366342656,"internally_reserved":0,"allocated":98304,"data_stored":24020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]}]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7652,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849948,"statfs":{"total":107374182400,"available":107366346752,"internally_reserved":0,"allocated":94208,"data_stored":23342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T17:28:05.220 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-24T17:28:05.221 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-01-24T17:28:05.221 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-01-24T17:28:05.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-01-24T17:28:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:05.594 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-01-24T17:28:05.606 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-01-24T17:28:05.607 INFO:teuthology.run_tasks:Running task vault... 2023-01-24T17:28:05.619 DEBUG:tasks.vault:Vault config is {'client.0': {'engine': 'transit', 'flavor': 'old', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/transit/keys/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1'}, {'path': 'my-key-2'}]}} 2023-01-24T17:28:05.619 INFO:tasks.vault:Downloading Vault... 2023-01-24T17:28:05.620 INFO:tasks.vault:Downloading Vault... 2023-01-24T17:28:05.620 DEBUG:teuthology.orchestra.run.smithi077:> curl -L https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip -o /home/ubuntu/cephtest/vault.zip 2023-01-24T17:28:05.652 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:28:05.652 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:28:06.146 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 42.6M 100 42.6M 0 0 86.2M 0 --:--:-- --:--:-- --:--:-- 86.2M 2023-01-24T17:28:06.147 INFO:tasks.vault:Verifying SHA256 signature... 2023-01-24T17:28:06.147 DEBUG:teuthology.orchestra.run.smithi077:> echo '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 /home/ubuntu/cephtest/vault.zip' | sha256sum --check --status 2023-01-24T17:28:06.323 INFO:tasks.vault:Extracting vault... 2023-01-24T17:28:06.323 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p /home/ubuntu/cephtest/vault 2023-01-24T17:28:06.341 DEBUG:teuthology.orchestra.run.smithi077:> unzip /home/ubuntu/cephtest/vault.zip -d /home/ubuntu/cephtest/vault 2023-01-24T17:28:06.414 INFO:teuthology.orchestra.run.smithi077.stdout:Archive: /home/ubuntu/cephtest/vault.zip 2023-01-24T17:28:07.140 INFO:teuthology.orchestra.run.smithi077.stdout: inflating: /home/ubuntu/cephtest/vault/vault 2023-01-24T17:28:07.140 INFO:tasks.vault:Starting Vault listening on 0.0.0.0:8200 ... 2023-01-24T17:28:07.141 INFO:tasks.vault.client.0:Restarting daemon 2023-01-24T17:28:07.141 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/vault && exec bash -c 'chmod +x /home/ubuntu/cephtest/vault/vault && /home/ubuntu/cephtest/vault/vault server -dev -dev-listen-address=0.0.0.0:8200 -dev-no-store-token -dev-root-token-id=test_root_token' & { read; kill %1; }) 2023-01-24T17:28:07.183 INFO:tasks.vault.client.0:Started 2023-01-24T17:28:07.230 INFO:tasks.vault.client.0.smithi077.stdout:==> Vault server configuration: 2023-01-24T17:28:07.230 INFO:tasks.vault.client.0.smithi077.stdout: 2023-01-24T17:28:07.230 INFO:tasks.vault.client.0.smithi077.stdout: Api Address: http://0.0.0.0:8200 2023-01-24T17:28:07.231 INFO:tasks.vault.client.0.smithi077.stdout: Cgo: disabled 2023-01-24T17:28:07.231 INFO:tasks.vault.client.0.smithi077.stdout: Cluster Address: https://0.0.0.0:8201 2023-01-24T17:28:07.234 INFO:tasks.vault.client.0.smithi077.stdout: Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled") 2023-01-24T17:28:07.234 INFO:tasks.vault.client.0.smithi077.stdout: Log Level: info 2023-01-24T17:28:07.234 INFO:tasks.vault.client.0.smithi077.stdout: Mlock: supported: true, enabled: false 2023-01-24T17:28:07.234 INFO:tasks.vault.client.0.smithi077.stdout: Storage: inmem 2023-01-24T17:28:07.234 INFO:tasks.vault.client.0.smithi077.stdout: Version: Vault v1.2.2 2023-01-24T17:28:07.235 INFO:tasks.vault.client.0.smithi077.stdout: 2023-01-24T17:28:07.249 INFO:tasks.vault.client.0.smithi077.stdout:WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory 2023-01-24T17:28:07.249 INFO:tasks.vault.client.0.smithi077.stdout:and starts unsealed with a single unseal key. The root token is already 2023-01-24T17:28:07.249 INFO:tasks.vault.client.0.smithi077.stdout:authenticated to the CLI, so you can immediately begin using Vault. 2023-01-24T17:28:07.250 INFO:tasks.vault.client.0.smithi077.stdout: 2023-01-24T17:28:07.250 INFO:tasks.vault.client.0.smithi077.stdout:You may need to set the following environment variable: 2023-01-24T17:28:07.250 INFO:tasks.vault.client.0.smithi077.stdout: 2023-01-24T17:28:07.250 INFO:tasks.vault.client.0.smithi077.stdout: $ export VAULT_ADDR='http://0.0.0.0:8200' 2023-01-24T17:28:07.250 INFO:tasks.vault.client.0.smithi077.stdout: 2023-01-24T17:28:07.251 INFO:tasks.vault.client.0.smithi077.stdout:The unseal key and root token are displayed below in case you want to 2023-01-24T17:28:07.251 INFO:tasks.vault.client.0.smithi077.stdout:seal/unseal the Vault or re-authenticate. 2023-01-24T17:28:07.251 INFO:tasks.vault.client.0.smithi077.stdout: 2023-01-24T17:28:07.251 INFO:tasks.vault.client.0.smithi077.stdout:Unseal Key: zJyOhyaO2qC/A75tEKVwnHPSPYcWA5QtQYBCesm9OKw= 2023-01-24T17:28:07.251 INFO:tasks.vault.client.0.smithi077.stdout:Root Token: test_root_token 2023-01-24T17:28:07.252 INFO:tasks.vault.client.0.smithi077.stdout: 2023-01-24T17:28:07.252 INFO:tasks.vault.client.0.smithi077.stdout:Development mode should NOT be used in production installations! 2023-01-24T17:28:07.252 INFO:tasks.vault.client.0.smithi077.stdout: 2023-01-24T17:28:07.252 INFO:tasks.vault.client.0.smithi077.stdout:==> Vault server started! Log data will stream in below: 2023-01-24T17:28:07.252 INFO:tasks.vault.client.0.smithi077.stdout: 2023-01-24T17:28:07.253 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.227Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set 2023-01-24T17:28:07.253 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.227Z [INFO] core: security barrier not initialized 2023-01-24T17:28:07.253 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.227Z [INFO] core: security barrier initialized: shares=1 threshold=1 2023-01-24T17:28:07.254 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.228Z [INFO] core: post-unseal setup starting 2023-01-24T17:28:07.254 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.235Z [INFO] core: loaded wrapping token key 2023-01-24T17:28:07.254 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.235Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2023-01-24T17:28:07.254 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.235Z [INFO] core: no mounts; adding default mount table 2023-01-24T17:28:07.254 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.236Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2023-01-24T17:28:07.255 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.237Z [INFO] core: successfully mounted backend: type=system path=sys/ 2023-01-24T17:28:07.255 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.237Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2023-01-24T17:28:07.255 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.238Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2023-01-24T17:28:07.255 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.238Z [INFO] core: restoring leases 2023-01-24T17:28:07.255 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.238Z [INFO] rollback: starting rollback manager 2023-01-24T17:28:07.256 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.238Z [INFO] expiration: lease restore complete 2023-01-24T17:28:07.256 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.238Z [INFO] identity: entities restored 2023-01-24T17:28:07.256 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.238Z [INFO] identity: groups restored 2023-01-24T17:28:07.256 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.238Z [INFO] core: post-unseal setup complete 2023-01-24T17:28:07.256 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.239Z [INFO] core: root token generated 2023-01-24T17:28:07.257 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.239Z [INFO] core: pre-seal teardown starting 2023-01-24T17:28:07.257 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.239Z [INFO] rollback: stopping rollback manager 2023-01-24T17:28:07.257 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.239Z [INFO] core: pre-seal teardown complete 2023-01-24T17:28:07.257 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.239Z [INFO] core.cluster-listener: starting listener: listener_address=0.0.0.0:8201 2023-01-24T17:28:07.257 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.239Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201 2023-01-24T17:28:07.257 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.239Z [INFO] core: post-unseal setup starting 2023-01-24T17:28:07.258 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.239Z [INFO] core: loaded wrapping token key 2023-01-24T17:28:07.258 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.239Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2023-01-24T17:28:07.258 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.239Z [INFO] core: successfully mounted backend: type=system path=sys/ 2023-01-24T17:28:07.258 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.240Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2023-01-24T17:28:07.258 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.240Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2023-01-24T17:28:07.259 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.240Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2023-01-24T17:28:07.259 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.240Z [INFO] core: restoring leases 2023-01-24T17:28:07.259 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.240Z [INFO] rollback: starting rollback manager 2023-01-24T17:28:07.259 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.240Z [INFO] identity: entities restored 2023-01-24T17:28:07.259 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.240Z [INFO] identity: groups restored 2023-01-24T17:28:07.260 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.240Z [INFO] core: post-unseal setup complete 2023-01-24T17:28:07.260 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.240Z [INFO] core: vault is unsealed 2023-01-24T17:28:07.260 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.240Z [INFO] expiration: lease restore complete 2023-01-24T17:28:07.260 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.241Z [INFO] expiration: revoked lease: lease_id=auth/token/root/h057b5783fa5937702842105aeb185f1cdabc13ac08261b3986c4240a96c619cc 2023-01-24T17:28:07.260 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.242Z [INFO] core: successful mount: namespace= path=secret/ type=kv 2023-01-24T17:28:07.261 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.252Z [INFO] secrets.kv.kv_fe080c3d: collecting keys to upgrade 2023-01-24T17:28:07.261 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.252Z [INFO] secrets.kv.kv_fe080c3d: done collecting keys: num_keys=1 2023-01-24T17:28:07.261 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:07.252Z [INFO] secrets.kv.kv_fe080c3d: upgrading keys finished 2023-01-24T17:28:17.184 INFO:tasks.vault:Mounting transit secrets engine 2023-01-24T17:28:17.185 INFO:tasks.vault:Send request to Vault: smithi077.front.sepia.ceph.com:8200 at /v1/sys/mounts/transit with token: test_root_token 2023-01-24T17:28:17.191 INFO:tasks.vault:b'' 2023-01-24T17:28:17.191 INFO:tasks.vault:Send request to Vault: smithi077.front.sepia.ceph.com:8200 at /v1/transit/keys/my-key-1 with token: test_root_token 2023-01-24T17:28:17.192 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:28:17.189Z [INFO] core: successful mount: namespace= path=transit/ type=transit 2023-01-24T17:28:17.194 INFO:tasks.vault:b'' 2023-01-24T17:28:17.194 INFO:tasks.vault:Send request to Vault: smithi077.front.sepia.ceph.com:8200 at /v1/transit/keys/my-key-2 with token: test_root_token 2023-01-24T17:28:17.196 INFO:tasks.vault:b'' 2023-01-24T17:28:17.197 INFO:tasks.vault:secrets created 2023-01-24T17:28:17.197 INFO:teuthology.run_tasks:Running task rgw... 2023-01-24T17:28:17.211 DEBUG:tasks.rgw:config is {'client.0': {'use-vault-role': 'client.0'}} 2023-01-24T17:28:17.211 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-01-24T17:28:17.212 INFO:tasks.rgw:Creating data pools 2023-01-24T17:28:17.212 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-01-24T17:28:17.212 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2023-01-24T17:28:17.600 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'default.rgw.buckets.data' created 2023-01-24T17:28:17.614 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-01-24T17:28:18.615 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-01-24T17:28:18.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-01-24T17:28:19.617 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'default.rgw.buckets.index' created 2023-01-24T17:28:19.633 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-01-24T17:28:20.621 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-01-24T17:28:20.636 DEBUG:tasks.rgw:Pools created 2023-01-24T17:28:20.637 INFO:tasks.rgw:Starting rgw... 2023-01-24T17:28:20.637 INFO:tasks.rgw:rgw client.0 config is {'use-vault-role': 'client.0'} 2023-01-24T17:28:20.637 INFO:tasks.rgw:Using beast as radosgw frontend 2023-01-24T17:28:20.638 DEBUG:teuthology.orchestra.run.smithi077:> sudo echo -n test_root_token | sudo tee /etc/ceph/vault-root-token 2023-01-24T17:28:20.666 INFO:teuthology.orchestra.run.smithi077.stdout:test_root_token 2023-01-24T17:28:20.666 INFO:tasks.rgw:Token file content 2023-01-24T17:28:20.667 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/ceph/vault-root-token 2023-01-24T17:28:20.724 INFO:teuthology.orchestra.run.smithi077.stdout:test_root_token 2023-01-24T17:28:20.725 INFO:tasks.rgw:Restrict access to token file 2023-01-24T17:28:20.725 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 600 /etc/ceph/vault-root-token 2023-01-24T17:28:20.791 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown ceph /etc/ceph/vault-root-token 2023-01-24T17:28:20.859 INFO:tasks.rgw.client.0:Restarting daemon 2023-01-24T17:28:20.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi077.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --rgw_crypt_sse_s3_vault_addr smithi077.front.sepia.ceph.com:8200 --rgw_crypt_sse_s3_vault_token_file /etc/ceph/vault-root-token --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-01-24T17:28:20.901 INFO:tasks.rgw.client.0:Started 2023-01-24T17:28:20.902 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi077.front.sepia.ceph.com:80/ 2023-01-24T17:28:20.903 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:80/ 2023-01-24T17:28:20.924 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:28:20.924 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:28:20.926 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T17:28:20.926 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi077.front.sepia.ceph.com port 80: Connection refused 2023-01-24T17:28:21.927 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:80/ 2023-01-24T17:28:21.954 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T17:28:21.955 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:28:21.955 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:28:21.955 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi077.front.sepia.ceph.com port 80: Connection refused 2023-01-24T17:28:23.956 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:80/ 2023-01-24T17:28:23.983 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:28:23.984 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:28:23.985 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T17:28:23.986 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi077.front.sepia.ceph.com port 80: Connection refused 2023-01-24T17:28:27.987 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:80/ 2023-01-24T17:28:28.013 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:28:28.014 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:28:28.015 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T17:28:28.016 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi077.front.sepia.ceph.com port 80: Connection refused 2023-01-24T17:28:36.018 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:80/ 2023-01-24T17:28:36.045 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:28:36.045 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:28:36.046 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-01-24T17:28:36.047 INFO:teuthology.orchestra.run.smithi077.stdout:anonymous 2023-01-24T17:28:36.047 INFO:teuthology.run_tasks:Running task s3tests... 2023-01-24T17:28:36.062 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}} 2023-01-24T17:28:36.063 INFO:tasks.s3tests:Downloading s3-tests... 2023-01-24T17:28:36.063 INFO:tasks.s3tests:Using branch 'ceph-quincy' for s3tests 2023-01-24T17:28:36.064 DEBUG:teuthology.orchestra.run.smithi077:> git clone -b ceph-quincy https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests 2023-01-24T17:28:36.108 INFO:teuthology.orchestra.run.smithi077.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests'... 2023-01-24T17:28:36.524 INFO:tasks.s3tests:Creating rgw users... 2023-01-24T17:28:36.525 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2023-01-24T17:28:36.526 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key MKGTKFEOGFFSSUHDNLRG --secret HpzCvCqD+NEKq2pflULGzrGXp6IOS2VkEd0jg/JSeOddRxIozn34iQ== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-01-24T17:28:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:28:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:28:36.622 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.619+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.626 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.623+0000 7f0410cd4680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.626 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.623+0000 7f0410cd4680 20 realm 2023-01-24T17:28:36.626 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.623+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.626 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.623+0000 7f0410cd4680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.627 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.623+0000 7f0410cd4680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:28:36.627 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.623+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.627 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.624+0000 7f0410cd4680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.627 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.624+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.628 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.624+0000 7f0410cd4680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:36.628 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.624+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.628 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.624+0000 7f0410cd4680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:28:36.629 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.624+0000 7f0410cd4680 20 searching for the correct realm 2023-01-24T17:28:36.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.632+0000 7f0410cd4680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:28:36.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.632+0000 7f0410cd4680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:28:36.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.632+0000 7f0410cd4680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:28:36.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.632+0000 7f0410cd4680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:28:36.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.632+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.632+0000 7f0410cd4680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.632+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.633+0000 7f0410cd4680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:36.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.633+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.633+0000 7f0410cd4680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:28:36.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.633+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.633+0000 7f0410cd4680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.633+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.633+0000 7f0410cd4680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.633+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.634+0000 7f0410cd4680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:36.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.640+0000 7f0410cd4680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:28:36.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.640+0000 7f0410cd4680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:28:36.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.640+0000 7f0410cd4680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:28:36.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.640+0000 7f0410cd4680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:28:36.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.640+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.640+0000 7f0410cd4680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:36.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.640+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.640+0000 7f0410cd4680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:28:36.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.640+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.640+0000 7f0410cd4680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:36.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.640+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:28:36.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 zone default found 2023-01-24T17:28:36.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 4 Realm: () 2023-01-24T17:28:36.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 4 ZoneGroup: default (0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f) 2023-01-24T17:28:36.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 4 Zone: default (97c5572f-f788-4cd6-8a34-3994e797b864) 2023-01-24T17:28:36.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:28:36.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 zonegroup default 2023-01-24T17:28:36.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 started sync module instance, tier type = 2023-01-24T17:28:36.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.641+0000 7f0410cd4680 20 started zone id=97c5572f-f788-4cd6-8a34-3994e797b864 (name=default) with tier type = 2023-01-24T17:28:36.677 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.674+0000 7f0410cd4680 20 add_watcher() i=0 2023-01-24T17:28:36.677 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.674+0000 7f0410cd4680 20 add_watcher() i=1 2023-01-24T17:28:36.678 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.674+0000 7f0410cd4680 20 add_watcher() i=2 2023-01-24T17:28:36.678 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.674+0000 7f0410cd4680 20 add_watcher() i=3 2023-01-24T17:28:36.678 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.674+0000 7f0410cd4680 20 add_watcher() i=4 2023-01-24T17:28:36.678 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.674+0000 7f0410cd4680 20 add_watcher() i=5 2023-01-24T17:28:36.678 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.674+0000 7f0410cd4680 20 add_watcher() i=6 2023-01-24T17:28:36.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.677+0000 7f0410cd4680 20 add_watcher() i=7 2023-01-24T17:28:36.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.677+0000 7f0410cd4680 2 all 8 watchers are set, enabling cache 2023-01-24T17:28:36.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.680+0000 7f03d57e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:28:36.683 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.680+0000 7f0410cd4680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:28:36.683 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.680+0000 7f0410cd4680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:28:36.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.684+0000 7f0410cd4680 5 note: GC not initialized 2023-01-24T17:28:36.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.685+0000 7f0343fff700 20 reqs_thread_entry: start 2023-01-24T17:28:36.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.685+0000 7f0341ffb700 20 reqs_thread_entry: start 2023-01-24T17:28:36.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.685+0000 7f0410cd4680 20 init_complete bucket index max shards: 11 2023-01-24T17:28:36.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.685+0000 7f0410cd4680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:28:36.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.685+0000 7f032ffff700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:29:06 2023 2023-01-24T17:28:36.690 INFO:teuthology.orchestra.run.smithi077.stderr: 2023-01-24T17:28:36.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.685+0000 7f032f7fe700 20 reqs_thread_entry: start 2023-01-24T17:28:36.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.685+0000 7f0410cd4680 20 get_system_obj_state: rctx=0x7ffe5084fe48 obj=default.rgw.meta:users.uid:foo.client.0 state=0x5596f1048940 s->prefetch_data=0 2023-01-24T17:28:36.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.685+0000 7f0410cd4680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-01-24T17:28:36.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2023-01-24T17:28:36.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:28:36.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 20 get_system_obj_state: rctx=0x7ffe5084fb38 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x5596f1048940 s->prefetch_data=0 2023-01-24T17:28:36.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2023-01-24T17:28:36.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2023-01-24T17:28:36.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-01-24T17:28:36.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 20 get_system_obj_state: rctx=0x7ffe5084fb38 obj=default.rgw.meta:users.keys:MKGTKFEOGFFSSUHDNLRG state=0x5596f1048940 s->prefetch_data=0 2023-01-24T17:28:36.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 10 cache get: name=default.rgw.meta+users.keys+MKGTKFEOGFFSSUHDNLRG : miss 2023-01-24T17:28:36.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 10 cache put: name=default.rgw.meta+users.keys+MKGTKFEOGFFSSUHDNLRG info.flags=0x0 2023-01-24T17:28:36.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 10 adding default.rgw.meta+users.keys+MKGTKFEOGFFSSUHDNLRG to cache LRU end 2023-01-24T17:28:36.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 20 get_system_obj_state: rctx=0x7ffe5084f3c8 obj=default.rgw.meta:users.keys:MKGTKFEOGFFSSUHDNLRG state=0x5596f1048940 s->prefetch_data=0 2023-01-24T17:28:36.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 10 cache get: name=default.rgw.meta+users.keys+MKGTKFEOGFFSSUHDNLRG : hit (negative entry) 2023-01-24T17:28:36.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 20 get_system_obj_state: rctx=0x7ffe5084fa78 obj=default.rgw.meta:users.keys:MKGTKFEOGFFSSUHDNLRG state=0x5596f1048940 s->prefetch_data=0 2023-01-24T17:28:36.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.686+0000 7f0410cd4680 10 cache get: name=default.rgw.meta+users.keys+MKGTKFEOGFFSSUHDNLRG : hit (negative entry) 2023-01-24T17:28:36.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.687+0000 7f0410cd4680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T17:28:36.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.687+0000 7f0410cd4680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:28:36.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.687+0000 7f0410cd4680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-01-24T17:28:36.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.688+0000 7f04025b9700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149696 cookie 94106778652320 notifier 4208 bl.length()=640 2023-01-24T17:28:36.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.688+0000 7f04025b9700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T17:28:36.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.688+0000 7f04025b9700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:28:36.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.690+0000 7f0410cd4680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-01-24T17:28:36.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.690+0000 7f0410cd4680 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-01-24T17:28:36.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.690+0000 7f0410cd4680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-01-24T17:28:36.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.690+0000 7f04025b9700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149696 cookie 94106778680464 notifier 4208 bl.length()=194 2023-01-24T17:28:36.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.690+0000 7f04025b9700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-01-24T17:28:36.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.690+0000 7f04025b9700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-01-24T17:28:36.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.693+0000 7f0410cd4680 10 cache put: name=default.rgw.meta+users.keys+MKGTKFEOGFFSSUHDNLRG info.flags=0x7 2023-01-24T17:28:36.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.693+0000 7f0410cd4680 10 moving default.rgw.meta+users.keys+MKGTKFEOGFFSSUHDNLRG to cache LRU end 2023-01-24T17:28:36.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.693+0000 7f0410cd4680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.keys:MKGTKFEOGFFSSUHDNLRG, ofs0, ns] 2023-01-24T17:28:36.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.693+0000 7f0401db8700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149697 cookie 94106778652320 notifier 4208 bl.length()=186 2023-01-24T17:28:36.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.693+0000 7f0401db8700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+MKGTKFEOGFFSSUHDNLRG info.flags=0x7 2023-01-24T17:28:36.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.693+0000 7f0401db8700 10 rgw watcher librados: moving default.rgw.meta+users.keys+MKGTKFEOGFFSSUHDNLRG to cache LRU end 2023-01-24T17:28:36.700 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-01-24T17:28:36.700 INFO:teuthology.orchestra.run.smithi077.stdout: "user_id": "foo.client.0", 2023-01-24T17:28:36.700 INFO:teuthology.orchestra.run.smithi077.stdout: "display_name": "Mr. foo.client.0", 2023-01-24T17:28:36.700 INFO:teuthology.orchestra.run.smithi077.stdout: "email": "foo.client.0+test@test.test", 2023-01-24T17:28:36.700 INFO:teuthology.orchestra.run.smithi077.stdout: "suspended": 0, 2023-01-24T17:28:36.701 INFO:teuthology.orchestra.run.smithi077.stdout: "max_buckets": 1000, 2023-01-24T17:28:36.701 INFO:teuthology.orchestra.run.smithi077.stdout: "subusers": [], 2023-01-24T17:28:36.701 INFO:teuthology.orchestra.run.smithi077.stdout: "keys": [ 2023-01-24T17:28:36.701 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-01-24T17:28:36.702 INFO:teuthology.orchestra.run.smithi077.stdout: "user": "foo.client.0", 2023-01-24T17:28:36.702 INFO:teuthology.orchestra.run.smithi077.stdout: "access_key": "MKGTKFEOGFFSSUHDNLRG", 2023-01-24T17:28:36.702 INFO:teuthology.orchestra.run.smithi077.stdout: "secret_key": "HpzCvCqD+NEKq2pflULGzrGXp6IOS2VkEd0jg/JSeOddRxIozn34iQ==" 2023-01-24T17:28:36.702 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-01-24T17:28:36.702 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2023-01-24T17:28:36.703 INFO:teuthology.orchestra.run.smithi077.stdout: "swift_keys": [], 2023-01-24T17:28:36.703 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": [ 2023-01-24T17:28:36.703 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-01-24T17:28:36.703 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "user-policy", 2023-01-24T17:28:36.703 INFO:teuthology.orchestra.run.smithi077.stdout: "perm": "*" 2023-01-24T17:28:36.704 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-01-24T17:28:36.704 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2023-01-24T17:28:36.704 INFO:teuthology.orchestra.run.smithi077.stdout: "op_mask": "read, write, delete", 2023-01-24T17:28:36.704 INFO:teuthology.orchestra.run.smithi077.stdout: "default_placement": "", 2023-01-24T17:28:36.705 INFO:teuthology.orchestra.run.smithi077.stdout: "default_storage_class": "", 2023-01-24T17:28:36.705 INFO:teuthology.orchestra.run.smithi077.stdout: "placement_tags": [], 2023-01-24T17:28:36.705 INFO:teuthology.orchestra.run.smithi077.stdout: "bucket_quota": { 2023-01-24T17:28:36.705 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": false, 2023-01-24T17:28:36.705 INFO:teuthology.orchestra.run.smithi077.stdout: "check_on_raw": false, 2023-01-24T17:28:36.706 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size": -1, 2023-01-24T17:28:36.706 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size_kb": 0, 2023-01-24T17:28:36.706 INFO:teuthology.orchestra.run.smithi077.stdout: "max_objects": -1 2023-01-24T17:28:36.706 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-01-24T17:28:36.706 INFO:teuthology.orchestra.run.smithi077.stdout: "user_quota": { 2023-01-24T17:28:36.707 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": false, 2023-01-24T17:28:36.707 INFO:teuthology.orchestra.run.smithi077.stdout: "check_on_raw": false, 2023-01-24T17:28:36.707 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size": -1, 2023-01-24T17:28:36.707 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size_kb": 0, 2023-01-24T17:28:36.708 INFO:teuthology.orchestra.run.smithi077.stdout: "max_objects": -1 2023-01-24T17:28:36.708 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-01-24T17:28:36.708 INFO:teuthology.orchestra.run.smithi077.stdout: "temp_url_keys": [], 2023-01-24T17:28:36.708 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "rgw", 2023-01-24T17:28:36.708 INFO:teuthology.orchestra.run.smithi077.stdout: "mfa_ids": [] 2023-01-24T17:28:36.709 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-01-24T17:28:36.709 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:36.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.694+0000 7f0410cd4680 20 remove_watcher() i=0 2023-01-24T17:28:36.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.694+0000 7f0410cd4680 2 removed watcher, disabling cache 2023-01-24T17:28:36.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.696+0000 7f0410cd4680 20 remove_watcher() i=1 2023-01-24T17:28:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.699+0000 7f0410cd4680 20 remove_watcher() i=2 2023-01-24T17:28:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.701+0000 7f0410cd4680 20 remove_watcher() i=3 2023-01-24T17:28:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.704+0000 7f0410cd4680 20 remove_watcher() i=4 2023-01-24T17:28:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.704+0000 7f0410cd4680 20 remove_watcher() i=5 2023-01-24T17:28:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.707+0000 7f0410cd4680 20 remove_watcher() i=6 2023-01-24T17:28:36.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.708+0000 7f0410cd4680 20 remove_watcher() i=7 2023-01-24T17:28:36.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.709+0000 7f03d57e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:28:36.716 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 2905556804 --totp-seed 4S47N7NNTQADYMUX6D7LUG5OG53AJ3TD2BA5TZRFDETTJ7A5IF3P3WBCTTQI2PYH --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-01-24T17:28:36.768 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:28:36.768 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:28:36.774 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.772+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.775 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.773+0000 7fcd4db56680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.773+0000 7fcd4db56680 20 realm 2023-01-24T17:28:36.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.773+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.773+0000 7fcd4db56680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.773+0000 7fcd4db56680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:28:36.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.773+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.773+0000 7fcd4db56680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.773+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.778 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.773+0000 7fcd4db56680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:36.778 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.773+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.778 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.774+0000 7fcd4db56680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:28:36.779 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.774+0000 7fcd4db56680 20 searching for the correct realm 2023-01-24T17:28:36.784 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:28:36.784 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:28:36.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:28:36.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:28:36.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.786 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.786 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:36.786 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.786 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:28:36.786 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.787 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.787 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.781+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.787 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.782+0000 7fcd4db56680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.787 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.782+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.787 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.782+0000 7fcd4db56680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:36.792 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.788+0000 7fcd4db56680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:28:36.792 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.788+0000 7fcd4db56680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:28:36.792 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.788+0000 7fcd4db56680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:28:36.792 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.788+0000 7fcd4db56680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:28:36.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.788+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.788+0000 7fcd4db56680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:36.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.788+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.788+0000 7fcd4db56680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:28:36.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.788+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.788+0000 7fcd4db56680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:36.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.788+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:28:36.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 zone default found 2023-01-24T17:28:36.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 4 Realm: () 2023-01-24T17:28:36.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 4 ZoneGroup: default (0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f) 2023-01-24T17:28:36.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 4 Zone: default (97c5572f-f788-4cd6-8a34-3994e797b864) 2023-01-24T17:28:36.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:28:36.796 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 zonegroup default 2023-01-24T17:28:36.796 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.796 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.796 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.797 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.797 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.797 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:36.797 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 started sync module instance, tier type = 2023-01-24T17:28:36.797 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.789+0000 7fcd4db56680 20 started zone id=97c5572f-f788-4cd6-8a34-3994e797b864 (name=default) with tier type = 2023-01-24T17:28:36.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.816+0000 7fcd4db56680 20 add_watcher() i=0 2023-01-24T17:28:36.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.816+0000 7fcd4db56680 20 add_watcher() i=1 2023-01-24T17:28:36.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.816+0000 7fcd4db56680 20 add_watcher() i=2 2023-01-24T17:28:36.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.816+0000 7fcd4db56680 20 add_watcher() i=3 2023-01-24T17:28:36.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.816+0000 7fcd4db56680 20 add_watcher() i=4 2023-01-24T17:28:36.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.816+0000 7fcd4db56680 20 add_watcher() i=5 2023-01-24T17:28:36.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.816+0000 7fcd4db56680 20 add_watcher() i=6 2023-01-24T17:28:36.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.818+0000 7fcd4db56680 20 add_watcher() i=7 2023-01-24T17:28:36.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.818+0000 7fcd4db56680 2 all 8 watchers are set, enabling cache 2023-01-24T17:28:36.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.821+0000 7fcd127e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:28:36.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.821+0000 7fcd4db56680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:28:36.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.821+0000 7fcd4db56680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:28:36.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.825+0000 7fcd4db56680 5 note: GC not initialized 2023-01-24T17:28:36.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.826+0000 7fcc817fa700 20 reqs_thread_entry: start 2023-01-24T17:28:36.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.826+0000 7fcc7affd700 20 reqs_thread_entry: start 2023-01-24T17:28:36.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcd4db56680 20 init_complete bucket index max shards: 11 2023-01-24T17:28:36.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcd4db56680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:28:36.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcc797fa700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:29:07 2023 2023-01-24T17:28:36.830 INFO:teuthology.orchestra.run.smithi077.stderr: 2023-01-24T17:28:36.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcc78ff9700 20 reqs_thread_entry: start 2023-01-24T17:28:36.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcd4db56680 20 get_system_obj_state: rctx=0x7ffc06fce8b8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55639c183940 s->prefetch_data=0 2023-01-24T17:28:36.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcd4db56680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-01-24T17:28:36.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcd4db56680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-01-24T17:28:36.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcd4db56680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:28:36.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcd4db56680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:28:36.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcd4db56680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:28:36.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcd4db56680 20 Read xattr: user.rgw.idtag 2023-01-24T17:28:36.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcd4db56680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:28:36.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.827+0000 7fcd4db56680 20 rados->read ofs=0 len=0 2023-01-24T17:28:36.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.828+0000 7fcd4db56680 20 rados_obj.operate() r=0 bl.length=455 2023-01-24T17:28:36.834 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.828+0000 7fcd4db56680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-01-24T17:28:36.834 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.828+0000 7fcd4db56680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:28:36.834 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:36.828+0000 7fcd4db56680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:28:38.434 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.432+0000 7fcd4db56680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T17:28:38.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.432+0000 7fcd4db56680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:28:38.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.432+0000 7fcd4db56680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-01-24T17:28:38.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.432+0000 7fcd3e439700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084290 cookie 93886310554400 notifier 4214 bl.length()=654 2023-01-24T17:28:38.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.432+0000 7fcd3e439700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T17:28:38.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.432+0000 7fcd3e439700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:28:38.436 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.434+0000 7fcd4db56680 20 remove_watcher() i=0 2023-01-24T17:28:38.436 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.434+0000 7fcd4db56680 2 removed watcher, disabling cache 2023-01-24T17:28:38.437 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.435+0000 7fcd4db56680 20 remove_watcher() i=1 2023-01-24T17:28:38.438 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.436+0000 7fcd4db56680 20 remove_watcher() i=2 2023-01-24T17:28:38.439 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.437+0000 7fcd4db56680 20 remove_watcher() i=3 2023-01-24T17:28:38.440 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.438+0000 7fcd4db56680 20 remove_watcher() i=4 2023-01-24T17:28:38.441 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.439+0000 7fcd4db56680 20 remove_watcher() i=5 2023-01-24T17:28:38.442 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.440+0000 7fcd4db56680 20 remove_watcher() i=6 2023-01-24T17:28:38.443 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.441+0000 7fcd4db56680 20 remove_watcher() i=7 2023-01-24T17:28:38.444 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.442+0000 7fcd127e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:28:38.450 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2023-01-24T17:28:38.450 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key BUQJDKRXKEPYNHRODVNF --secret cUAGZpmykwknJSWDe9+SA/p8y+Deo/7yVXvG/OJouMVpewIuITRC1g== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-01-24T17:28:38.505 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:28:38.505 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:28:38.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.508+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.509+0000 7feb0b9b1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.509+0000 7feb0b9b1680 20 realm 2023-01-24T17:28:38.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.509+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.510+0000 7feb0b9b1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.510+0000 7feb0b9b1680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:28:38.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.510+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.510+0000 7feb0b9b1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.514 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.510+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.514 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.510+0000 7feb0b9b1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.510+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.511+0000 7feb0b9b1680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:28:38.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.511+0000 7feb0b9b1680 20 searching for the correct realm 2023-01-24T17:28:38.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.518+0000 7feb0b9b1680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:28:38.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.518+0000 7feb0b9b1680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:28:38.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.518+0000 7feb0b9b1680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:28:38.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.518+0000 7feb0b9b1680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:28:38.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.518+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.518+0000 7feb0b9b1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.518+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.518+0000 7feb0b9b1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.518+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.523 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.519+0000 7feb0b9b1680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:28:38.523 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.519+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.519+0000 7feb0b9b1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.519+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.519+0000 7feb0b9b1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.519+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.519+0000 7feb0b9b1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.529 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.525+0000 7feb0b9b1680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:28:38.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.525+0000 7feb0b9b1680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:28:38.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.525+0000 7feb0b9b1680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:28:38.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.525+0000 7feb0b9b1680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:28:38.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.525+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.526+0000 7feb0b9b1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.526+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.526+0000 7feb0b9b1680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:28:38.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.526+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.526+0000 7feb0b9b1680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.526+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.526+0000 7feb0b9b1680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:28:38.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.526+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 20 zone default found 2023-01-24T17:28:38.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 4 Realm: () 2023-01-24T17:28:38.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 4 ZoneGroup: default (0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f) 2023-01-24T17:28:38.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 4 Zone: default (97c5572f-f788-4cd6-8a34-3994e797b864) 2023-01-24T17:28:38.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:28:38.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 20 zonegroup default 2023-01-24T17:28:38.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.534 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.534 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.534 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.534 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 20 started sync module instance, tier type = 2023-01-24T17:28:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.527+0000 7feb0b9b1680 20 started zone id=97c5572f-f788-4cd6-8a34-3994e797b864 (name=default) with tier type = 2023-01-24T17:28:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.543+0000 7feb0b9b1680 20 add_watcher() i=0 2023-01-24T17:28:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.543+0000 7feb0b9b1680 20 add_watcher() i=1 2023-01-24T17:28:38.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.543+0000 7feb0b9b1680 20 add_watcher() i=2 2023-01-24T17:28:38.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.543+0000 7feb0b9b1680 20 add_watcher() i=3 2023-01-24T17:28:38.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.543+0000 7feb0b9b1680 20 add_watcher() i=4 2023-01-24T17:28:38.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.543+0000 7feb0b9b1680 20 add_watcher() i=5 2023-01-24T17:28:38.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.543+0000 7feb0b9b1680 20 add_watcher() i=6 2023-01-24T17:28:38.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.544+0000 7feb0b9b1680 20 add_watcher() i=7 2023-01-24T17:28:38.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.544+0000 7feb0b9b1680 2 all 8 watchers are set, enabling cache 2023-01-24T17:28:38.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.545+0000 7fea4bfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:28:38.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.545+0000 7feb0b9b1680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:28:38.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.545+0000 7feb0b9b1680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:28:38.551 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.549+0000 7feb0b9b1680 5 note: GC not initialized 2023-01-24T17:28:38.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.549+0000 7fea42ffd700 20 reqs_thread_entry: start 2023-01-24T17:28:38.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.550+0000 7fea40ff9700 20 reqs_thread_entry: start 2023-01-24T17:28:38.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.550+0000 7feb0b9b1680 20 init_complete bucket index max shards: 11 2023-01-24T17:28:38.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.550+0000 7feb0b9b1680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:28:38.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.550+0000 7fea2effd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:29:08 2023 2023-01-24T17:28:38.560 INFO:teuthology.orchestra.run.smithi077.stderr: 2023-01-24T17:28:38.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.550+0000 7fea2e7fc700 20 reqs_thread_entry: start 2023-01-24T17:28:38.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.550+0000 7feb0b9b1680 20 get_system_obj_state: rctx=0x7ffdb64b0e88 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55bceff30940 s->prefetch_data=0 2023-01-24T17:28:38.561 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.550+0000 7feb0b9b1680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-01-24T17:28:38.561 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.550+0000 7feb0b9b1680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2023-01-24T17:28:38.561 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.550+0000 7feb0b9b1680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:28:38.561 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.550+0000 7feb0b9b1680 20 get_system_obj_state: rctx=0x7ffdb64b0b78 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x55bceff30940 s->prefetch_data=0 2023-01-24T17:28:38.561 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.550+0000 7feb0b9b1680 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2023-01-24T17:28:38.562 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.551+0000 7feb0b9b1680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2023-01-24T17:28:38.562 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.551+0000 7feb0b9b1680 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-01-24T17:28:38.562 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.551+0000 7feb0b9b1680 20 get_system_obj_state: rctx=0x7ffdb64b0b78 obj=default.rgw.meta:users.keys:BUQJDKRXKEPYNHRODVNF state=0x55bceff30940 s->prefetch_data=0 2023-01-24T17:28:38.562 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.551+0000 7feb0b9b1680 10 cache get: name=default.rgw.meta+users.keys+BUQJDKRXKEPYNHRODVNF : miss 2023-01-24T17:28:38.562 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.551+0000 7feb0b9b1680 10 cache put: name=default.rgw.meta+users.keys+BUQJDKRXKEPYNHRODVNF info.flags=0x0 2023-01-24T17:28:38.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.551+0000 7feb0b9b1680 10 adding default.rgw.meta+users.keys+BUQJDKRXKEPYNHRODVNF to cache LRU end 2023-01-24T17:28:38.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.551+0000 7feb0b9b1680 20 get_system_obj_state: rctx=0x7ffdb64b0408 obj=default.rgw.meta:users.keys:BUQJDKRXKEPYNHRODVNF state=0x55bceff30940 s->prefetch_data=0 2023-01-24T17:28:38.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.551+0000 7feb0b9b1680 10 cache get: name=default.rgw.meta+users.keys+BUQJDKRXKEPYNHRODVNF : hit (negative entry) 2023-01-24T17:28:38.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.551+0000 7feb0b9b1680 20 get_system_obj_state: rctx=0x7ffdb64b0ab8 obj=default.rgw.meta:users.keys:BUQJDKRXKEPYNHRODVNF state=0x55bceff30940 s->prefetch_data=0 2023-01-24T17:28:38.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.551+0000 7feb0b9b1680 10 cache get: name=default.rgw.meta+users.keys+BUQJDKRXKEPYNHRODVNF : hit (negative entry) 2023-01-24T17:28:38.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.552+0000 7feb0b9b1680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T17:28:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.552+0000 7feb0b9b1680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:28:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.552+0000 7feb0b9b1680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-01-24T17:28:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.552+0000 7feafd296700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084289 cookie 94269969501472 notifier 4220 bl.length()=640 2023-01-24T17:28:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.552+0000 7feafd296700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T17:28:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.552+0000 7feafd296700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:28:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.554+0000 7feb0b9b1680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-01-24T17:28:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.554+0000 7feb0b9b1680 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-01-24T17:28:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.554+0000 7feb0b9b1680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-01-24T17:28:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.554+0000 7feafd296700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084290 cookie 94269969501472 notifier 4220 bl.length()=194 2023-01-24T17:28:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.554+0000 7feafd296700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-01-24T17:28:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.554+0000 7feafd296700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-01-24T17:28:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.556+0000 7feb0b9b1680 10 cache put: name=default.rgw.meta+users.keys+BUQJDKRXKEPYNHRODVNF info.flags=0x7 2023-01-24T17:28:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.556+0000 7feb0b9b1680 10 moving default.rgw.meta+users.keys+BUQJDKRXKEPYNHRODVNF to cache LRU end 2023-01-24T17:28:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.556+0000 7feb0b9b1680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:BUQJDKRXKEPYNHRODVNF, ofs0, ns] 2023-01-24T17:28:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.556+0000 7feafca95700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084291 cookie 94269969512848 notifier 4220 bl.length()=186 2023-01-24T17:28:38.567 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.556+0000 7feafca95700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+BUQJDKRXKEPYNHRODVNF info.flags=0x7 2023-01-24T17:28:38.567 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.556+0000 7feafca95700 10 rgw watcher librados: moving default.rgw.meta+users.keys+BUQJDKRXKEPYNHRODVNF to cache LRU end 2023-01-24T17:28:38.567 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-01-24T17:28:38.568 INFO:teuthology.orchestra.run.smithi077.stdout: "user_id": "bar.client.0", 2023-01-24T17:28:38.568 INFO:teuthology.orchestra.run.smithi077.stdout: "display_name": "Mr. bar.client.0", 2023-01-24T17:28:38.568 INFO:teuthology.orchestra.run.smithi077.stdout: "email": "bar.client.0+test@test.test", 2023-01-24T17:28:38.568 INFO:teuthology.orchestra.run.smithi077.stdout: "suspended": 0, 2023-01-24T17:28:38.568 INFO:teuthology.orchestra.run.smithi077.stdout: "max_buckets": 1000, 2023-01-24T17:28:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "subusers": [], 2023-01-24T17:28:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "keys": [ 2023-01-24T17:28:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-01-24T17:28:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "user": "bar.client.0", 2023-01-24T17:28:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "access_key": "BUQJDKRXKEPYNHRODVNF", 2023-01-24T17:28:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "secret_key": "cUAGZpmykwknJSWDe9+SA/p8y+Deo/7yVXvG/OJouMVpewIuITRC1g==" 2023-01-24T17:28:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-01-24T17:28:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2023-01-24T17:28:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "swift_keys": [], 2023-01-24T17:28:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": [ 2023-01-24T17:28:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-01-24T17:28:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "user-policy", 2023-01-24T17:28:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "perm": "*" 2023-01-24T17:28:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-01-24T17:28:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2023-01-24T17:28:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "op_mask": "read, write, delete", 2023-01-24T17:28:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "default_placement": "", 2023-01-24T17:28:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "default_storage_class": "", 2023-01-24T17:28:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "placement_tags": [], 2023-01-24T17:28:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "bucket_quota": { 2023-01-24T17:28:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": false, 2023-01-24T17:28:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "check_on_raw": false, 2023-01-24T17:28:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size": -1, 2023-01-24T17:28:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size_kb": 0, 2023-01-24T17:28:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "max_objects": -1 2023-01-24T17:28:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-01-24T17:28:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "user_quota": { 2023-01-24T17:28:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": false, 2023-01-24T17:28:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "check_on_raw": false, 2023-01-24T17:28:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size": -1, 2023-01-24T17:28:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size_kb": 0, 2023-01-24T17:28:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "max_objects": -1 2023-01-24T17:28:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-01-24T17:28:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "temp_url_keys": [], 2023-01-24T17:28:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "rgw", 2023-01-24T17:28:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "mfa_ids": [] 2023-01-24T17:28:38.576 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-01-24T17:28:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:38.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.558+0000 7feb0b9b1680 20 remove_watcher() i=0 2023-01-24T17:28:38.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.558+0000 7feb0b9b1680 2 removed watcher, disabling cache 2023-01-24T17:28:38.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.559+0000 7feb0b9b1680 20 remove_watcher() i=1 2023-01-24T17:28:38.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.559+0000 7feb0b9b1680 20 remove_watcher() i=2 2023-01-24T17:28:38.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.562+0000 7feb0b9b1680 20 remove_watcher() i=3 2023-01-24T17:28:38.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.565+0000 7feb0b9b1680 20 remove_watcher() i=4 2023-01-24T17:28:38.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.566+0000 7feb0b9b1680 20 remove_watcher() i=5 2023-01-24T17:28:38.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.566+0000 7feb0b9b1680 20 remove_watcher() i=6 2023-01-24T17:28:38.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.567+0000 7feb0b9b1680 20 remove_watcher() i=7 2023-01-24T17:28:38.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.568+0000 7fea4bfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:28:38.580 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 6119503861 --totp-seed 4IUCXNUZJM2OU2NYGSGGPVWKSD5JLF5O2HG5PTN33R6HHEAAKRRSVQTHZXD7X5ZF --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-01-24T17:28:38.632 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:28:38.632 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:28:38.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.636+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.637+0000 7f56b7d6e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.637+0000 7f56b7d6e680 20 realm 2023-01-24T17:28:38.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.637+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.637+0000 7f56b7d6e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.637+0000 7f56b7d6e680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:28:38.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.637+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.637+0000 7f56b7d6e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.637+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.638+0000 7f56b7d6e680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.638+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.638+0000 7f56b7d6e680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:28:38.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.638+0000 7f56b7d6e680 20 searching for the correct realm 2023-01-24T17:28:38.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.645+0000 7f56b7d6e680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:28:38.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.645+0000 7f56b7d6e680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:28:38.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.645+0000 7f56b7d6e680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:28:38.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.645+0000 7f56b7d6e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:28:38.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.645+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.645+0000 7f56b7d6e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.645+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.645+0000 7f56b7d6e680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.645+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.646+0000 7f56b7d6e680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:28:38.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.646+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.646+0000 7f56b7d6e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.646+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.646+0000 7f56b7d6e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.646+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.646+0000 7f56b7d6e680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.652+0000 7f56b7d6e680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:28:38.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.652+0000 7f56b7d6e680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:28:38.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.652+0000 7f56b7d6e680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:28:38.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.652+0000 7f56b7d6e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:28:38.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.652+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:28:38.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:28:38.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 20 zone default found 2023-01-24T17:28:38.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 4 Realm: () 2023-01-24T17:28:38.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 4 ZoneGroup: default (0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f) 2023-01-24T17:28:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 4 Zone: default (97c5572f-f788-4cd6-8a34-3994e797b864) 2023-01-24T17:28:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:28:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 20 zonegroup default 2023-01-24T17:28:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.653+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.654+0000 7f56b7d6e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.654+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.654+0000 7f56b7d6e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.654+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.654+0000 7f56b7d6e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.654+0000 7f56b7d6e680 20 started sync module instance, tier type = 2023-01-24T17:28:38.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.654+0000 7f56b7d6e680 20 started zone id=97c5572f-f788-4cd6-8a34-3994e797b864 (name=default) with tier type = 2023-01-24T17:28:38.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.678+0000 7f56b7d6e680 20 add_watcher() i=0 2023-01-24T17:28:38.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.678+0000 7f56b7d6e680 20 add_watcher() i=1 2023-01-24T17:28:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.678+0000 7f56b7d6e680 20 add_watcher() i=2 2023-01-24T17:28:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.678+0000 7f56b7d6e680 20 add_watcher() i=3 2023-01-24T17:28:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.678+0000 7f56b7d6e680 20 add_watcher() i=4 2023-01-24T17:28:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.678+0000 7f56b7d6e680 20 add_watcher() i=5 2023-01-24T17:28:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.678+0000 7f56b7d6e680 20 add_watcher() i=6 2023-01-24T17:28:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.681+0000 7f56b7d6e680 20 add_watcher() i=7 2023-01-24T17:28:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.681+0000 7f56b7d6e680 2 all 8 watchers are set, enabling cache 2023-01-24T17:28:38.685 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.682+0000 7f567bfe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:28:38.685 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.682+0000 7f56b7d6e680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:28:38.685 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.682+0000 7f56b7d6e680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:28:38.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.686+0000 7f56b7d6e680 5 note: GC not initialized 2023-01-24T17:28:38.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.687+0000 7f55eaffd700 20 reqs_thread_entry: start 2023-01-24T17:28:38.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.687+0000 7f55e8ff9700 20 reqs_thread_entry: start 2023-01-24T17:28:38.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.687+0000 7f56b7d6e680 20 init_complete bucket index max shards: 11 2023-01-24T17:28:38.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.687+0000 7f56b7d6e680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:28:38.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.687+0000 7f55deffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:29:09 2023 2023-01-24T17:28:38.691 INFO:teuthology.orchestra.run.smithi077.stderr: 2023-01-24T17:28:38.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.687+0000 7f55de7fc700 20 reqs_thread_entry: start 2023-01-24T17:28:38.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.687+0000 7f56b7d6e680 20 get_system_obj_state: rctx=0x7ffce7a30098 obj=default.rgw.meta:users.uid:bar.client.0 state=0x560e5ce20940 s->prefetch_data=0 2023-01-24T17:28:38.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.687+0000 7f56b7d6e680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-01-24T17:28:38.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.688+0000 7f56b7d6e680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-01-24T17:28:38.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.688+0000 7f56b7d6e680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:28:38.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.688+0000 7f56b7d6e680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:28:38.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.688+0000 7f56b7d6e680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:28:38.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.688+0000 7f56b7d6e680 20 Read xattr: user.rgw.idtag 2023-01-24T17:28:38.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.688+0000 7f56b7d6e680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:28:38.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.688+0000 7f56b7d6e680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.689+0000 7f56b7d6e680 20 rados_obj.operate() r=0 bl.length=455 2023-01-24T17:28:38.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.689+0000 7f56b7d6e680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-01-24T17:28:38.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.689+0000 7f56b7d6e680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:28:38.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.689+0000 7f56b7d6e680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:28:38.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.691+0000 7f56b7d6e680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T17:28:38.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.691+0000 7f56b7d6e680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:28:38.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.691+0000 7f56b7d6e680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-01-24T17:28:38.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.691+0000 7f56a8e52700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084291 cookie 94619689453504 notifier 4226 bl.length()=654 2023-01-24T17:28:38.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.691+0000 7f56a8e52700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T17:28:38.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.691+0000 7f56a8e52700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:28:38.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.692+0000 7f56b7d6e680 20 remove_watcher() i=0 2023-01-24T17:28:38.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.692+0000 7f56b7d6e680 2 removed watcher, disabling cache 2023-01-24T17:28:38.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.693+0000 7f56b7d6e680 20 remove_watcher() i=1 2023-01-24T17:28:38.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.695+0000 7f56b7d6e680 20 remove_watcher() i=2 2023-01-24T17:28:38.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.698+0000 7f56b7d6e680 20 remove_watcher() i=3 2023-01-24T17:28:38.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.701+0000 7f56b7d6e680 20 remove_watcher() i=4 2023-01-24T17:28:38.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.701+0000 7f56b7d6e680 20 remove_watcher() i=5 2023-01-24T17:28:38.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.703+0000 7f56b7d6e680 20 remove_watcher() i=6 2023-01-24T17:28:38.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.707+0000 7f56b7d6e680 20 remove_watcher() i=7 2023-01-24T17:28:38.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.707+0000 7f567bfe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:28:38.714 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2023-01-24T17:28:38.715 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key PKYDOPQFLRQMFLXCTRCV --secret IUB9AK7XzYFW42DdG5jqZPEFE4x5e34ddhx7HdOGGYwPhh8SPIuCoQ== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2023-01-24T17:28:38.767 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:28:38.768 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:28:38.773 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.771+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.774 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.772+0000 7f9f6e0df680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.775 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.772+0000 7f9f6e0df680 20 realm 2023-01-24T17:28:38.775 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.772+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.775 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.772+0000 7f9f6e0df680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.775 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.772+0000 7f9f6e0df680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:28:38.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.772+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.772+0000 7f9f6e0df680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.772+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.772+0000 7f9f6e0df680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.772+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.773+0000 7f9f6e0df680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:28:38.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.773+0000 7f9f6e0df680 20 searching for the correct realm 2023-01-24T17:28:38.783 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:28:38.783 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:28:38.783 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:28:38.784 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:28:38.784 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.784 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.784 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.784 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:28:38.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.780+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.786 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.781+0000 7f9f6e0df680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.786 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.781+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.786 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.781+0000 7f9f6e0df680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.790 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.786+0000 7f9f6e0df680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:28:38.790 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.786+0000 7f9f6e0df680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:28:38.790 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.786+0000 7f9f6e0df680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:28:38.790 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.786+0000 7f9f6e0df680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:28:38.790 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.786+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.791 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.787+0000 7f9f6e0df680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.791 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.787+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.791 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.787+0000 7f9f6e0df680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:28:38.791 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.787+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.791 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.787+0000 7f9f6e0df680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.792 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.787+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.792 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.787+0000 7f9f6e0df680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:28:38.792 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.787+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 20 zone default found 2023-01-24T17:28:38.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 4 Realm: () 2023-01-24T17:28:38.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 4 ZoneGroup: default (0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f) 2023-01-24T17:28:38.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 4 Zone: default (97c5572f-f788-4cd6-8a34-3994e797b864) 2023-01-24T17:28:38.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:28:38.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 20 zonegroup default 2023-01-24T17:28:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.796 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 20 started sync module instance, tier type = 2023-01-24T17:28:38.796 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.788+0000 7f9f6e0df680 20 started zone id=97c5572f-f788-4cd6-8a34-3994e797b864 (name=default) with tier type = 2023-01-24T17:28:38.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.799+0000 7f9f6e0df680 20 add_watcher() i=0 2023-01-24T17:28:38.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.799+0000 7f9f6e0df680 20 add_watcher() i=1 2023-01-24T17:28:38.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.799+0000 7f9f6e0df680 20 add_watcher() i=2 2023-01-24T17:28:38.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.799+0000 7f9f6e0df680 20 add_watcher() i=3 2023-01-24T17:28:38.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.799+0000 7f9f6e0df680 20 add_watcher() i=4 2023-01-24T17:28:38.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.799+0000 7f9f6e0df680 20 add_watcher() i=5 2023-01-24T17:28:38.803 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.799+0000 7f9f6e0df680 20 add_watcher() i=6 2023-01-24T17:28:38.803 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.800+0000 7f9f6e0df680 20 add_watcher() i=7 2023-01-24T17:28:38.804 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.800+0000 7f9f6e0df680 2 all 8 watchers are set, enabling cache 2023-01-24T17:28:38.804 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.801+0000 7f9f327e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:28:38.804 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.801+0000 7f9f6e0df680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:28:38.804 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.801+0000 7f9f6e0df680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:28:38.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.805+0000 7f9f6e0df680 5 note: GC not initialized 2023-01-24T17:28:38.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.806+0000 7f9ea17fa700 20 reqs_thread_entry: start 2023-01-24T17:28:38.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.806+0000 7f9e9affd700 20 reqs_thread_entry: start 2023-01-24T17:28:38.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.806+0000 7f9f6e0df680 20 init_complete bucket index max shards: 11 2023-01-24T17:28:38.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.806+0000 7f9f6e0df680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:28:38.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.806+0000 7f9e997fa700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:29:09 2023 2023-01-24T17:28:38.811 INFO:teuthology.orchestra.run.smithi077.stderr: 2023-01-24T17:28:38.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.806+0000 7f9e98ff9700 20 reqs_thread_entry: start 2023-01-24T17:28:38.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.806+0000 7f9f6e0df680 20 get_system_obj_state: rctx=0x7fffa2386438 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x56410c4c8940 s->prefetch_data=0 2023-01-24T17:28:38.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.806+0000 7f9f6e0df680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-01-24T17:28:38.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2023-01-24T17:28:38.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:28:38.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 20 get_system_obj_state: rctx=0x7fffa2386128 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x56410c4c8940 s->prefetch_data=0 2023-01-24T17:28:38.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2023-01-24T17:28:38.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2023-01-24T17:28:38.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-01-24T17:28:38.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 20 get_system_obj_state: rctx=0x7fffa2386128 obj=default.rgw.meta:users.keys:PKYDOPQFLRQMFLXCTRCV state=0x56410c4c8940 s->prefetch_data=0 2023-01-24T17:28:38.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 10 cache get: name=default.rgw.meta+users.keys+PKYDOPQFLRQMFLXCTRCV : miss 2023-01-24T17:28:38.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 10 cache put: name=default.rgw.meta+users.keys+PKYDOPQFLRQMFLXCTRCV info.flags=0x0 2023-01-24T17:28:38.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 10 adding default.rgw.meta+users.keys+PKYDOPQFLRQMFLXCTRCV to cache LRU end 2023-01-24T17:28:38.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 20 get_system_obj_state: rctx=0x7fffa23859b8 obj=default.rgw.meta:users.keys:PKYDOPQFLRQMFLXCTRCV state=0x56410c4c8940 s->prefetch_data=0 2023-01-24T17:28:38.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 10 cache get: name=default.rgw.meta+users.keys+PKYDOPQFLRQMFLXCTRCV : hit (negative entry) 2023-01-24T17:28:38.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 20 get_system_obj_state: rctx=0x7fffa2386068 obj=default.rgw.meta:users.keys:PKYDOPQFLRQMFLXCTRCV state=0x56410c4c8940 s->prefetch_data=0 2023-01-24T17:28:38.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.807+0000 7f9f6e0df680 10 cache get: name=default.rgw.meta+users.keys+PKYDOPQFLRQMFLXCTRCV : hit (negative entry) 2023-01-24T17:28:38.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.808+0000 7f9f6e0df680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T17:28:38.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.808+0000 7f9f6e0df680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:28:38.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.808+0000 7f9f6e0df680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-01-24T17:28:38.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.808+0000 7f9f5b7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084292 cookie 94837380819856 notifier 4232 bl.length()=714 2023-01-24T17:28:38.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.808+0000 7f9f5b7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T17:28:38.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.808+0000 7f9f5b7fe700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:28:38.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.810+0000 7f9f6e0df680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-01-24T17:28:38.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.810+0000 7f9f6e0df680 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-01-24T17:28:38.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.810+0000 7f9f6e0df680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-01-24T17:28:38.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.810+0000 7f9f5b7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084293 cookie 94837380818320 notifier 4232 bl.length()=224 2023-01-24T17:28:38.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.810+0000 7f9f5b7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-01-24T17:28:38.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.810+0000 7f9f5b7fe700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-01-24T17:28:38.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.812+0000 7f9f6e0df680 10 cache put: name=default.rgw.meta+users.keys+PKYDOPQFLRQMFLXCTRCV info.flags=0x7 2023-01-24T17:28:38.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.812+0000 7f9f6e0df680 10 moving default.rgw.meta+users.keys+PKYDOPQFLRQMFLXCTRCV to cache LRU end 2023-01-24T17:28:38.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.812+0000 7f9f6e0df680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.keys:PKYDOPQFLRQMFLXCTRCV, ofs0, ns] 2023-01-24T17:28:38.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.813+0000 7f9f5b7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084294 cookie 94837380818320 notifier 4232 bl.length()=201 2023-01-24T17:28:38.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.813+0000 7f9f5b7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+PKYDOPQFLRQMFLXCTRCV info.flags=0x7 2023-01-24T17:28:38.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.813+0000 7f9f5b7fe700 10 rgw watcher librados: moving default.rgw.meta+users.keys+PKYDOPQFLRQMFLXCTRCV to cache LRU end 2023-01-24T17:28:38.819 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-01-24T17:28:38.820 INFO:teuthology.orchestra.run.smithi077.stdout: "user_id": "testx$tenanteduser.client.0", 2023-01-24T17:28:38.820 INFO:teuthology.orchestra.run.smithi077.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2023-01-24T17:28:38.820 INFO:teuthology.orchestra.run.smithi077.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2023-01-24T17:28:38.820 INFO:teuthology.orchestra.run.smithi077.stdout: "suspended": 0, 2023-01-24T17:28:38.820 INFO:teuthology.orchestra.run.smithi077.stdout: "max_buckets": 1000, 2023-01-24T17:28:38.821 INFO:teuthology.orchestra.run.smithi077.stdout: "subusers": [], 2023-01-24T17:28:38.821 INFO:teuthology.orchestra.run.smithi077.stdout: "keys": [ 2023-01-24T17:28:38.821 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-01-24T17:28:38.821 INFO:teuthology.orchestra.run.smithi077.stdout: "user": "testx$tenanteduser.client.0", 2023-01-24T17:28:38.821 INFO:teuthology.orchestra.run.smithi077.stdout: "access_key": "PKYDOPQFLRQMFLXCTRCV", 2023-01-24T17:28:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: "secret_key": "IUB9AK7XzYFW42DdG5jqZPEFE4x5e34ddhx7HdOGGYwPhh8SPIuCoQ==" 2023-01-24T17:28:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-01-24T17:28:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2023-01-24T17:28:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: "swift_keys": [], 2023-01-24T17:28:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": [ 2023-01-24T17:28:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-01-24T17:28:38.823 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "user-policy", 2023-01-24T17:28:38.823 INFO:teuthology.orchestra.run.smithi077.stdout: "perm": "*" 2023-01-24T17:28:38.823 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-01-24T17:28:38.823 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2023-01-24T17:28:38.823 INFO:teuthology.orchestra.run.smithi077.stdout: "op_mask": "read, write, delete", 2023-01-24T17:28:38.823 INFO:teuthology.orchestra.run.smithi077.stdout: "default_placement": "", 2023-01-24T17:28:38.824 INFO:teuthology.orchestra.run.smithi077.stdout: "default_storage_class": "", 2023-01-24T17:28:38.824 INFO:teuthology.orchestra.run.smithi077.stdout: "placement_tags": [], 2023-01-24T17:28:38.824 INFO:teuthology.orchestra.run.smithi077.stdout: "bucket_quota": { 2023-01-24T17:28:38.824 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": false, 2023-01-24T17:28:38.824 INFO:teuthology.orchestra.run.smithi077.stdout: "check_on_raw": false, 2023-01-24T17:28:38.825 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size": -1, 2023-01-24T17:28:38.825 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size_kb": 0, 2023-01-24T17:28:38.825 INFO:teuthology.orchestra.run.smithi077.stdout: "max_objects": -1 2023-01-24T17:28:38.825 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-01-24T17:28:38.825 INFO:teuthology.orchestra.run.smithi077.stdout: "user_quota": { 2023-01-24T17:28:38.826 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": false, 2023-01-24T17:28:38.826 INFO:teuthology.orchestra.run.smithi077.stdout: "check_on_raw": false, 2023-01-24T17:28:38.826 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size": -1, 2023-01-24T17:28:38.826 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size_kb": 0, 2023-01-24T17:28:38.826 INFO:teuthology.orchestra.run.smithi077.stdout: "max_objects": -1 2023-01-24T17:28:38.826 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-01-24T17:28:38.827 INFO:teuthology.orchestra.run.smithi077.stdout: "temp_url_keys": [], 2023-01-24T17:28:38.827 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "rgw", 2023-01-24T17:28:38.827 INFO:teuthology.orchestra.run.smithi077.stdout: "mfa_ids": [] 2023-01-24T17:28:38.827 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-01-24T17:28:38.827 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:38.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.814+0000 7f9f6e0df680 20 remove_watcher() i=0 2023-01-24T17:28:38.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.814+0000 7f9f6e0df680 2 removed watcher, disabling cache 2023-01-24T17:28:38.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.815+0000 7f9f6e0df680 20 remove_watcher() i=1 2023-01-24T17:28:38.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.816+0000 7f9f6e0df680 20 remove_watcher() i=2 2023-01-24T17:28:38.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.817+0000 7f9f6e0df680 20 remove_watcher() i=3 2023-01-24T17:28:38.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.818+0000 7f9f6e0df680 20 remove_watcher() i=4 2023-01-24T17:28:38.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.819+0000 7f9f6e0df680 20 remove_watcher() i=5 2023-01-24T17:28:38.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.820+0000 7f9f6e0df680 20 remove_watcher() i=6 2023-01-24T17:28:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.820+0000 7f9f6e0df680 20 remove_watcher() i=7 2023-01-24T17:28:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.821+0000 7f9f327e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:28:38.831 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 3993757136 --totp-seed BGBZDFFUULQTMJDC55OS2POGAP4Z6ZMR2LSH3VH6EPCLWIAQWOFTGD7PAUL2DKSH --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-01-24T17:28:38.883 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:28:38.883 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:28:38.889 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.887+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.888+0000 7f25103fd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.888+0000 7f25103fd680 20 realm 2023-01-24T17:28:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.888+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.888+0000 7f25103fd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.888+0000 7f25103fd680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:28:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.888+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.888+0000 7f25103fd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.888+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.888+0000 7f25103fd680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.888+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.889+0000 7f25103fd680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:28:38.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.889+0000 7f25103fd680 20 searching for the correct realm 2023-01-24T17:28:38.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.896+0000 7f25103fd680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:28:38.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.896+0000 7f25103fd680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:28:38.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.896+0000 7f25103fd680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:28:38.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.896+0000 7f25103fd680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:28:38.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.896+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.896+0000 7f25103fd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.896+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.896+0000 7f25103fd680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.896+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.897+0000 7f25103fd680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:28:38.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.897+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.897+0000 7f25103fd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.897+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.902 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.897+0000 7f25103fd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.902 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.897+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.902 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.897+0000 7f25103fd680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.903+0000 7f25103fd680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:28:38.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.903+0000 7f25103fd680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:28:38.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.903+0000 7f25103fd680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:28:38.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.903+0000 7f25103fd680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:28:38.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.903+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.903+0000 7f25103fd680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.903+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:28:38.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.908 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:28:38.908 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.909 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:28:38.909 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.910 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.910 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 20 zone default found 2023-01-24T17:28:38.910 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 4 Realm: () 2023-01-24T17:28:38.910 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 4 ZoneGroup: default (0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f) 2023-01-24T17:28:38.910 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 4 Zone: default (97c5572f-f788-4cd6-8a34-3994e797b864) 2023-01-24T17:28:38.911 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:28:38.911 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 20 zonegroup default 2023-01-24T17:28:38.911 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.911 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.911 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.904+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.905+0000 7f25103fd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.905+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.905+0000 7f25103fd680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:28:38.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.905+0000 7f25103fd680 20 started sync module instance, tier type = 2023-01-24T17:28:38.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.905+0000 7f25103fd680 20 started zone id=97c5572f-f788-4cd6-8a34-3994e797b864 (name=default) with tier type = 2023-01-24T17:28:38.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.923+0000 7f25103fd680 20 add_watcher() i=0 2023-01-24T17:28:38.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.923+0000 7f25103fd680 20 add_watcher() i=1 2023-01-24T17:28:38.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.923+0000 7f25103fd680 20 add_watcher() i=2 2023-01-24T17:28:38.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.923+0000 7f25103fd680 20 add_watcher() i=3 2023-01-24T17:28:38.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.923+0000 7f25103fd680 20 add_watcher() i=4 2023-01-24T17:28:38.927 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.923+0000 7f25103fd680 20 add_watcher() i=5 2023-01-24T17:28:38.927 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.923+0000 7f25103fd680 20 add_watcher() i=6 2023-01-24T17:28:38.927 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.924+0000 7f25103fd680 20 add_watcher() i=7 2023-01-24T17:28:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.924+0000 7f25103fd680 2 all 8 watchers are set, enabling cache 2023-01-24T17:28:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.926+0000 7f244ffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:28:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.926+0000 7f25103fd680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:28:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.926+0000 7f25103fd680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:28:38.932 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.930+0000 7f25103fd680 5 note: GC not initialized 2023-01-24T17:28:38.934 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.930+0000 7f243effd700 20 reqs_thread_entry: start 2023-01-24T17:28:38.934 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f243cff9700 20 reqs_thread_entry: start 2023-01-24T17:28:38.934 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f25103fd680 20 init_complete bucket index max shards: 11 2023-01-24T17:28:38.934 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f25103fd680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:28:38.934 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f2436ffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:29:09 2023 2023-01-24T17:28:38.935 INFO:teuthology.orchestra.run.smithi077.stderr: 2023-01-24T17:28:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f24367fc700 20 reqs_thread_entry: start 2023-01-24T17:28:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f25103fd680 20 get_system_obj_state: rctx=0x7ffe898e7698 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55b50af57940 s->prefetch_data=0 2023-01-24T17:28:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f25103fd680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-01-24T17:28:38.935 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f25103fd680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-01-24T17:28:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f25103fd680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:28:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f25103fd680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:28:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f25103fd680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:28:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f25103fd680 20 Read xattr: user.rgw.idtag 2023-01-24T17:28:38.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f25103fd680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:28:38.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.931+0000 7f25103fd680 20 rados->read ofs=0 len=0 2023-01-24T17:28:38.938 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.932+0000 7f25103fd680 20 rados_obj.operate() r=0 bl.length=514 2023-01-24T17:28:38.938 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.932+0000 7f25103fd680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-01-24T17:28:38.938 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.932+0000 7f25103fd680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:28:38.938 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.932+0000 7f25103fd680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:28:38.938 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.934+0000 7f25103fd680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T17:28:38.939 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.934+0000 7f25103fd680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:28:38.939 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.934+0000 7f25103fd680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-01-24T17:28:38.939 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.934+0000 7f25014e1700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084295 cookie 94236062921216 notifier 4238 bl.length()=728 2023-01-24T17:28:38.939 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.934+0000 7f25014e1700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T17:28:38.939 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.934+0000 7f25014e1700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:28:38.940 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.936+0000 7f25103fd680 20 remove_watcher() i=0 2023-01-24T17:28:38.940 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.936+0000 7f25103fd680 2 removed watcher, disabling cache 2023-01-24T17:28:38.941 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.939+0000 7f25103fd680 20 remove_watcher() i=1 2023-01-24T17:28:38.942 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.939+0000 7f25103fd680 20 remove_watcher() i=2 2023-01-24T17:28:38.944 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.942+0000 7f25103fd680 20 remove_watcher() i=3 2023-01-24T17:28:38.947 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.945+0000 7f25103fd680 20 remove_watcher() i=4 2023-01-24T17:28:38.948 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.946+0000 7f25103fd680 20 remove_watcher() i=5 2023-01-24T17:28:38.951 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.949+0000 7f25103fd680 20 remove_watcher() i=6 2023-01-24T17:28:38.952 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.950+0000 7f25103fd680 20 remove_watcher() i=7 2023-01-24T17:28:38.953 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:28:38.950+0000 7f244ffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:28:38.957 INFO:tasks.s3tests:Configuring s3-tests... 2023-01-24T17:28:38.958 DEBUG:teuthology.orchestra.run.smithi077:> cd /home/ubuntu/cephtest/s3-tests && ./bootstrap 2023-01-24T17:28:47.740 INFO:teuthology.orchestra.run.smithi077.stderr:./bootstrap: Missing required RPM packages: libevent-devel libffi-devel libxml2-devel libxslt-devel zlib-devel. 2023-01-24T17:28:48.462 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:04:00 ago on Tue 24 Jan 2023 05:24:48 PM UTC. 2023-01-24T17:28:49.610 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-01-24T17:28:49.612 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:28:49.613 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-01-24T17:28:49.613 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:28:49.613 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-01-24T17:28:49.613 INFO:teuthology.orchestra.run.smithi077.stdout: libevent-devel x86_64 2.1.8-5.el8 CentOS-AppStream 104 k 2023-01-24T17:28:49.613 INFO:teuthology.orchestra.run.smithi077.stdout: libffi-devel x86_64 3.1-24.el8 CentOS-BaseOS 29 k 2023-01-24T17:28:49.614 INFO:teuthology.orchestra.run.smithi077.stdout: libxml2-devel x86_64 2.9.7-16.el8 CentOS-AppStream 1.0 M 2023-01-24T17:28:49.614 INFO:teuthology.orchestra.run.smithi077.stdout: libxslt-devel x86_64 1.1.32-6.el8 CentOS-AppStream 323 k 2023-01-24T17:28:49.614 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-21.el8 CentOS-BaseOS 58 k 2023-01-24T17:28:49.615 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2023-01-24T17:28:49.615 INFO:teuthology.orchestra.run.smithi077.stdout: libffi x86_64 3.1-24.el8 CentOS-BaseOS 38 k 2023-01-24T17:28:49.615 INFO:teuthology.orchestra.run.smithi077.stdout: libxml2 x86_64 2.9.7-16.el8 CentOS-BaseOS 696 k 2023-01-24T17:28:49.616 INFO:teuthology.orchestra.run.smithi077.stdout: python3-libxml2 x86_64 2.9.7-16.el8 CentOS-BaseOS 237 k 2023-01-24T17:28:49.616 INFO:teuthology.orchestra.run.smithi077.stdout: zlib x86_64 1.2.11-21.el8 CentOS-BaseOS 103 k 2023-01-24T17:28:49.616 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-01-24T17:28:49.617 INFO:teuthology.orchestra.run.smithi077.stdout: cmake-filesystem x86_64 3.20.2-4.el8 CentOS-AppStream 44 k 2023-01-24T17:28:49.618 INFO:teuthology.orchestra.run.smithi077.stdout: libgcrypt-devel x86_64 1.8.5-7.el8 CentOS-BaseOS 150 k 2023-01-24T17:28:49.618 INFO:teuthology.orchestra.run.smithi077.stdout: libgpg-error-devel x86_64 1.31-1.el8 CentOS-BaseOS 64 k 2023-01-24T17:28:49.618 INFO:teuthology.orchestra.run.smithi077.stdout: xz-devel x86_64 5.2.4-4.el8 CentOS-BaseOS 63 k 2023-01-24T17:28:49.618 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:49.618 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-01-24T17:28:49.619 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-01-24T17:28:49.619 INFO:teuthology.orchestra.run.smithi077.stdout:Install 9 Packages 2023-01-24T17:28:49.619 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 4 Packages 2023-01-24T17:28:49.619 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:49.620 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 2.9 M 2023-01-24T17:28:49.620 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-01-24T17:28:49.721 INFO:teuthology.orchestra.run.smithi077.stdout:(1/13): cmake-filesystem-3.20.2-4.el8.x86_64.rp 423 kB/s | 44 kB 00:00 2023-01-24T17:28:49.746 INFO:teuthology.orchestra.run.smithi077.stdout:(2/13): libevent-devel-2.1.8-5.el8.x86_64.rpm 803 kB/s | 104 kB 00:00 2023-01-24T17:28:49.771 INFO:teuthology.orchestra.run.smithi077.stdout:(3/13): libffi-devel-3.1-24.el8.x86_64.rpm 1.1 MB/s | 29 kB 00:00 2023-01-24T17:28:49.796 INFO:teuthology.orchestra.run.smithi077.stdout:(4/13): libxslt-devel-1.1.32-6.el8.x86_64.rpm 4.3 MB/s | 323 kB 00:00 2023-01-24T17:28:49.822 INFO:teuthology.orchestra.run.smithi077.stdout:(5/13): libgcrypt-devel-1.8.5-7.el8.x86_64.rpm 2.9 MB/s | 150 kB 00:00 2023-01-24T17:28:49.847 INFO:teuthology.orchestra.run.smithi077.stdout:(6/13): libgpg-error-devel-1.31-1.el8.x86_64.rp 1.2 MB/s | 64 kB 00:00 2023-01-24T17:28:49.872 INFO:teuthology.orchestra.run.smithi077.stdout:(7/13): xz-devel-5.2.4-4.el8.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-01-24T17:28:49.898 INFO:teuthology.orchestra.run.smithi077.stdout:(8/13): zlib-devel-1.2.11-21.el8.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2023-01-24T17:28:49.923 INFO:teuthology.orchestra.run.smithi077.stdout:(9/13): libffi-3.1-24.el8.x86_64.rpm 750 kB/s | 38 kB 00:00 2023-01-24T17:28:49.965 INFO:teuthology.orchestra.run.smithi077.stdout:(10/13): libxml2-2.9.7-16.el8.x86_64.rpm 10 MB/s | 696 kB 00:00 2023-01-24T17:28:49.990 INFO:teuthology.orchestra.run.smithi077.stdout:(11/13): python3-libxml2-2.9.7-16.el8.x86_64.rp 3.5 MB/s | 237 kB 00:00 2023-01-24T17:28:50.024 INFO:teuthology.orchestra.run.smithi077.stdout:(12/13): libxml2-devel-2.9.7-16.el8.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2023-01-24T17:28:50.049 INFO:teuthology.orchestra.run.smithi077.stdout:(13/13): zlib-1.2.11-21.el8.x86_64.rpm 1.2 MB/s | 103 kB 00:00 2023-01-24T17:28:50.052 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-01-24T17:28:50.052 INFO:teuthology.orchestra.run.smithi077.stdout:Total 6.6 MB/s | 2.9 MB 00:00 2023-01-24T17:28:50.169 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-01-24T17:28:50.311 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-01-24T17:28:50.312 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-01-24T17:28:50.456 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-01-24T17:28:50.460 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-01-24T17:28:51.056 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-01-24T17:28:51.316 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : zlib-1.2.11-21.el8.x86_64 1/17 2023-01-24T17:28:51.539 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libxml2-2.9.7-16.el8.x86_64 2/17 2023-01-24T17:28:51.715 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 3/17 2023-01-24T17:28:51.739 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 3/17 2023-01-24T17:28:51.923 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgcrypt-devel-1.8.5-7.el8.x86_64 4/17 2023-01-24T17:28:51.936 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libgcrypt-devel-1.8.5-7.el8.x86_64 4/17 2023-01-24T17:28:52.162 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 5/17 2023-01-24T17:28:52.312 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libffi-3.1-24.el8.x86_64 6/17 2023-01-24T17:28:52.490 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xz-devel-5.2.4-4.el8.x86_64 7/17 2023-01-24T17:28:52.911 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cmake-filesystem-3.20.2-4.el8.x86_64 8/17 2023-01-24T17:28:53.272 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libxml2-devel-2.9.7-16.el8.x86_64 9/17 2023-01-24T17:28:53.520 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 10/17 2023-01-24T17:28:53.640 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libffi-devel-3.1-24.el8.x86_64 11/17 2023-01-24T17:28:53.689 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libffi-devel-3.1-24.el8.x86_64 11/17 2023-01-24T17:28:53.910 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : python3-libxml2-2.9.7-16.el8.x86_64 12/17 2023-01-24T17:28:54.183 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 13/17 2023-01-24T17:28:54.332 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : python3-libxml2-2.9.7-14.el8.x86_64 14/17 2023-01-24T17:28:54.524 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libxml2-2.9.7-14.el8.x86_64 15/17 2023-01-24T17:28:54.682 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : zlib-1.2.11-19.el8.x86_64 16/17 2023-01-24T17:28:54.822 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libffi-3.1-23.el8.x86_64 17/17 2023-01-24T17:28:55.198 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libffi-3.1-23.el8.x86_64 17/17 2023-01-24T17:28:55.199 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cmake-filesystem-3.20.2-4.el8.x86_64 1/17 2023-01-24T17:28:55.199 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 2/17 2023-01-24T17:28:55.199 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libxml2-devel-2.9.7-16.el8.x86_64 3/17 2023-01-24T17:28:55.199 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 4/17 2023-01-24T17:28:55.200 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libffi-devel-3.1-24.el8.x86_64 5/17 2023-01-24T17:28:55.200 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgcrypt-devel-1.8.5-7.el8.x86_64 6/17 2023-01-24T17:28:55.200 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 7/17 2023-01-24T17:28:55.200 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xz-devel-5.2.4-4.el8.x86_64 8/17 2023-01-24T17:28:55.200 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 9/17 2023-01-24T17:28:55.201 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libffi-3.1-24.el8.x86_64 10/17 2023-01-24T17:28:55.201 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libffi-3.1-23.el8.x86_64 11/17 2023-01-24T17:28:55.201 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libxml2-2.9.7-16.el8.x86_64 12/17 2023-01-24T17:28:55.201 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libxml2-2.9.7-14.el8.x86_64 13/17 2023-01-24T17:28:55.201 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-libxml2-2.9.7-16.el8.x86_64 14/17 2023-01-24T17:28:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-libxml2-2.9.7-14.el8.x86_64 15/17 2023-01-24T17:28:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-1.2.11-21.el8.x86_64 16/17 2023-01-24T17:28:55.881 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-1.2.11-19.el8.x86_64 17/17 2023-01-24T17:28:55.881 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:55.882 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2023-01-24T17:28:55.882 INFO:teuthology.orchestra.run.smithi077.stdout: libffi-3.1-24.el8.x86_64 libxml2-2.9.7-16.el8.x86_64 2023-01-24T17:28:55.882 INFO:teuthology.orchestra.run.smithi077.stdout: python3-libxml2-2.9.7-16.el8.x86_64 zlib-1.2.11-21.el8.x86_64 2023-01-24T17:28:55.882 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-01-24T17:28:55.882 INFO:teuthology.orchestra.run.smithi077.stdout: cmake-filesystem-3.20.2-4.el8.x86_64 libevent-devel-2.1.8-5.el8.x86_64 2023-01-24T17:28:55.883 INFO:teuthology.orchestra.run.smithi077.stdout: libffi-devel-3.1-24.el8.x86_64 libgcrypt-devel-1.8.5-7.el8.x86_64 2023-01-24T17:28:55.883 INFO:teuthology.orchestra.run.smithi077.stdout: libgpg-error-devel-1.31-1.el8.x86_64 libxml2-devel-2.9.7-16.el8.x86_64 2023-01-24T17:28:55.883 INFO:teuthology.orchestra.run.smithi077.stdout: libxslt-devel-1.1.32-6.el8.x86_64 xz-devel-5.2.4-4.el8.x86_64 2023-01-24T17:28:55.883 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-21.el8.x86_64 2023-01-24T17:28:55.883 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:28:55.884 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-01-24T17:28:56.508 INFO:teuthology.orchestra.run.smithi077.stdout:Already using interpreter /usr/bin/python3.6 2023-01-24T17:28:56.508 INFO:teuthology.orchestra.run.smithi077.stdout:Using base prefix '/usr' 2023-01-24T17:28:56.508 INFO:teuthology.orchestra.run.smithi077.stdout:New python executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python3.6 2023-01-24T17:28:56.508 INFO:teuthology.orchestra.run.smithi077.stdout:Also creating executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python 2023-01-24T17:29:00.863 INFO:teuthology.orchestra.run.smithi077.stdout:Installing setuptools, pip, wheel...done. 2023-01-24T17:29:01.199 INFO:teuthology.orchestra.run.smithi077.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.6/site-packages (21.3.1) 2023-01-24T17:29:02.663 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting setuptools==44.1.0 2023-01-24T17:29:02.775 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2023-01-24T17:29:03.117 INFO:teuthology.orchestra.run.smithi077.stdout:Installing collected packages: setuptools 2023-01-24T17:29:03.117 INFO:teuthology.orchestra.run.smithi077.stdout: Attempting uninstall: setuptools 2023-01-24T17:29:03.117 INFO:teuthology.orchestra.run.smithi077.stdout: Found existing installation: setuptools 59.6.0 2023-01-24T17:29:03.164 INFO:teuthology.orchestra.run.smithi077.stdout: Uninstalling setuptools-59.6.0: 2023-01-24T17:29:03.173 INFO:teuthology.orchestra.run.smithi077.stdout: Successfully uninstalled setuptools-59.6.0 2023-01-24T17:29:03.393 INFO:teuthology.orchestra.run.smithi077.stdout:Successfully installed setuptools-44.1.0 2023-01-24T17:29:04.075 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting PyYAML 2023-01-24T17:29:04.163 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (603 kB) 2023-01-24T17:29:04.308 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting nose>=1.0.0 2023-01-24T17:29:04.325 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2023-01-24T17:29:04.438 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting boto>=2.6.0 2023-01-24T17:29:04.459 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2023-01-24T17:29:05.362 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting boto3>=1.0.0 2023-01-24T17:29:05.381 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2023-01-24T17:29:06.367 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting botocore<1.28.0 2023-01-24T17:29:06.386 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2023-01-24T17:29:06.621 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting munch>=2.0.0 2023-01-24T17:29:06.638 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading munch-2.5.0-py2.py3-none-any.whl (10 kB) 2023-01-24T17:29:07.016 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting gevent>=1.0 2023-01-24T17:29:07.035 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading gevent-22.10.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB) 2023-01-24T17:29:07.229 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting isodate>=0.4.4 2023-01-24T17:29:07.246 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2023-01-24T17:29:07.397 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting requests>=2.23.0 2023-01-24T17:29:07.414 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2023-01-24T17:29:07.606 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting pytz>=2011k 2023-01-24T17:29:07.624 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading pytz-2022.7.1-py2.py3-none-any.whl (499 kB) 2023-01-24T17:29:07.727 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting httplib2 2023-01-24T17:29:07.745 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading httplib2-0.21.0-py3-none-any.whl (96 kB) 2023-01-24T17:29:08.254 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting lxml 2023-01-24T17:29:08.273 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading lxml-4.9.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.6 MB) 2023-01-24T17:29:08.503 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2023-01-24T17:29:08.520 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2023-01-24T17:29:08.690 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting jmespath<2.0.0,>=0.7.1 2023-01-24T17:29:08.707 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2023-01-24T17:29:08.847 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting urllib3<1.27,>=1.25.4 2023-01-24T17:29:08.866 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB) 2023-01-24T17:29:08.949 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting python-dateutil<3.0.0,>=2.1 2023-01-24T17:29:08.966 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-01-24T17:29:09.062 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting six 2023-01-24T17:29:09.079 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-01-24T17:29:09.198 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting zope.event 2023-01-24T17:29:09.215 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB) 2023-01-24T17:29:09.254 INFO:teuthology.orchestra.run.smithi077.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (from gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2023-01-24T17:29:09.628 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting greenlet>=2.0.0 2023-01-24T17:29:09.647 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading greenlet-2.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (513 kB) 2023-01-24T17:29:10.056 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting zope.interface 2023-01-24T17:29:10.074 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253 kB) 2023-01-24T17:29:10.214 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting certifi>=2017.4.17 2023-01-24T17:29:10.232 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading certifi-2022.12.7-py3-none-any.whl (155 kB) 2023-01-24T17:29:10.324 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting idna<4,>=2.5 2023-01-24T17:29:10.341 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading idna-3.4-py3-none-any.whl (61 kB) 2023-01-24T17:29:10.536 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting charset-normalizer~=2.0.0 2023-01-24T17:29:10.553 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2023-01-24T17:29:10.716 INFO:teuthology.orchestra.run.smithi077.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2023-01-24T17:29:10.734 INFO:teuthology.orchestra.run.smithi077.stdout: Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB) 2023-01-24T17:29:11.113 INFO:teuthology.orchestra.run.smithi077.stdout:Installing collected packages: six, urllib3, python-dateutil, jmespath, botocore, zope.interface, zope.event, s3transfer, pyparsing, idna, greenlet, charset-normalizer, certifi, requests, PyYAML, pytz, nose, munch, lxml, isodate, httplib2, gevent, boto3, boto 2023-01-24T17:29:14.517 INFO:teuthology.orchestra.run.smithi077.stdout:Successfully installed PyYAML-6.0 boto-2.49.0 boto3-1.23.10 botocore-1.26.10 certifi-2022.12.7 charset-normalizer-2.0.12 gevent-22.10.2 greenlet-2.0.1 httplib2-0.21.0 idna-3.4 isodate-0.6.1 jmespath-0.10.0 lxml-4.9.2 munch-2.5.0 nose-1.3.7 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2022.7.1 requests-2.27.1 s3transfer-0.5.2 six-1.16.0 urllib3-1.26.14 zope.event-4.6 zope.interface-5.5.2 2023-01-24T17:29:14.820 INFO:teuthology.orchestra.run.smithi077.stdout:running develop 2023-01-24T17:29:14.840 INFO:teuthology.orchestra.run.smithi077.stdout:running egg_info 2023-01-24T17:29:14.840 INFO:teuthology.orchestra.run.smithi077.stdout:creating s3tests.egg-info 2023-01-24T17:29:14.840 INFO:teuthology.orchestra.run.smithi077.stdout:writing s3tests.egg-info/PKG-INFO 2023-01-24T17:29:14.841 INFO:teuthology.orchestra.run.smithi077.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2023-01-24T17:29:14.841 INFO:teuthology.orchestra.run.smithi077.stdout:writing requirements to s3tests.egg-info/requires.txt 2023-01-24T17:29:14.841 INFO:teuthology.orchestra.run.smithi077.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2023-01-24T17:29:14.841 INFO:teuthology.orchestra.run.smithi077.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-01-24T17:29:14.851 INFO:teuthology.orchestra.run.smithi077.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2023-01-24T17:29:14.851 INFO:teuthology.orchestra.run.smithi077.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-01-24T17:29:14.861 INFO:teuthology.orchestra.run.smithi077.stdout:running build_ext 2023-01-24T17:29:14.861 INFO:teuthology.orchestra.run.smithi077.stdout:Creating /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/s3tests.egg-link (link to .) 2023-01-24T17:29:14.862 INFO:teuthology.orchestra.run.smithi077.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2023-01-24T17:29:14.862 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.862 INFO:teuthology.orchestra.run.smithi077.stdout:Installed /home/ubuntu/cephtest/s3-tests 2023-01-24T17:29:14.863 INFO:teuthology.orchestra.run.smithi077.stdout:Processing dependencies for s3tests==0.0.1 2023-01-24T17:29:14.954 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for isodate==0.6.1 2023-01-24T17:29:14.955 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: isodate 0.6.1 2023-01-24T17:29:14.955 INFO:teuthology.orchestra.run.smithi077.stdout:Adding isodate 0.6.1 to easy-install.pth file 2023-01-24T17:29:14.955 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.956 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.956 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for gevent==22.10.2 2023-01-24T17:29:14.956 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: gevent 22.10.2 2023-01-24T17:29:14.956 INFO:teuthology.orchestra.run.smithi077.stdout:Adding gevent 22.10.2 to easy-install.pth file 2023-01-24T17:29:14.956 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.957 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.958 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for munch==2.5.0 2023-01-24T17:29:14.958 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: munch 2.5.0 2023-01-24T17:29:14.959 INFO:teuthology.orchestra.run.smithi077.stdout:Adding munch 2.5.0 to easy-install.pth file 2023-01-24T17:29:14.959 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.959 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.959 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for PyYAML==6.0 2023-01-24T17:29:14.960 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: PyYAML 6.0 2023-01-24T17:29:14.960 INFO:teuthology.orchestra.run.smithi077.stdout:Adding PyYAML 6.0 to easy-install.pth file 2023-01-24T17:29:14.960 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.960 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.961 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for boto3==1.23.10 2023-01-24T17:29:14.961 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: boto3 1.23.10 2023-01-24T17:29:14.968 INFO:teuthology.orchestra.run.smithi077.stdout:Adding boto3 1.23.10 to easy-install.pth file 2023-01-24T17:29:14.968 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.968 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.969 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for boto==2.49.0 2023-01-24T17:29:14.969 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: boto 2.49.0 2023-01-24T17:29:14.969 INFO:teuthology.orchestra.run.smithi077.stdout:Adding boto 2.49.0 to easy-install.pth file 2023-01-24T17:29:14.969 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.969 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.970 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for six==1.16.0 2023-01-24T17:29:14.970 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: six 1.16.0 2023-01-24T17:29:14.970 INFO:teuthology.orchestra.run.smithi077.stdout:Adding six 1.16.0 to easy-install.pth file 2023-01-24T17:29:14.970 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.970 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.971 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for zope.event==4.6 2023-01-24T17:29:14.971 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: zope.event 4.6 2023-01-24T17:29:14.971 INFO:teuthology.orchestra.run.smithi077.stdout:Adding zope.event 4.6 to easy-install.pth file 2023-01-24T17:29:14.971 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.971 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.972 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for setuptools==44.1.0 2023-01-24T17:29:14.972 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: setuptools 44.1.0 2023-01-24T17:29:14.972 INFO:teuthology.orchestra.run.smithi077.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2023-01-24T17:29:14.972 INFO:teuthology.orchestra.run.smithi077.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2023-01-24T17:29:14.973 INFO:teuthology.orchestra.run.smithi077.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2023-01-24T17:29:14.973 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.973 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.973 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for greenlet==2.0.1 2023-01-24T17:29:14.973 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: greenlet 2.0.1 2023-01-24T17:29:14.974 INFO:teuthology.orchestra.run.smithi077.stdout:Adding greenlet 2.0.1 to easy-install.pth file 2023-01-24T17:29:14.974 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.974 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.974 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for zope.interface==5.5.2 2023-01-24T17:29:14.974 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: zope.interface 5.5.2 2023-01-24T17:29:14.975 INFO:teuthology.orchestra.run.smithi077.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2023-01-24T17:29:14.975 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.975 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.975 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for jmespath==0.10.0 2023-01-24T17:29:14.975 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: jmespath 0.10.0 2023-01-24T17:29:14.976 INFO:teuthology.orchestra.run.smithi077.stdout:Adding jmespath 0.10.0 to easy-install.pth file 2023-01-24T17:29:14.976 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.976 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.976 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for s3transfer==0.5.2 2023-01-24T17:29:14.977 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: s3transfer 0.5.2 2023-01-24T17:29:14.977 INFO:teuthology.orchestra.run.smithi077.stdout:Adding s3transfer 0.5.2 to easy-install.pth file 2023-01-24T17:29:14.977 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.978 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.978 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for botocore==1.26.10 2023-01-24T17:29:14.978 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: botocore 1.26.10 2023-01-24T17:29:14.979 INFO:teuthology.orchestra.run.smithi077.stdout:Adding botocore 1.26.10 to easy-install.pth file 2023-01-24T17:29:14.979 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.979 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.980 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for urllib3==1.26.14 2023-01-24T17:29:14.980 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: urllib3 1.26.14 2023-01-24T17:29:14.980 INFO:teuthology.orchestra.run.smithi077.stdout:Adding urllib3 1.26.14 to easy-install.pth file 2023-01-24T17:29:14.981 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.981 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.982 INFO:teuthology.orchestra.run.smithi077.stdout:Searching for python-dateutil==2.8.2 2023-01-24T17:29:14.982 INFO:teuthology.orchestra.run.smithi077.stdout:Best match: python-dateutil 2.8.2 2023-01-24T17:29:14.982 INFO:teuthology.orchestra.run.smithi077.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2023-01-24T17:29:14.983 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:29:14.983 INFO:teuthology.orchestra.run.smithi077.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2023-01-24T17:29:14.983 INFO:teuthology.orchestra.run.smithi077.stdout:Finished processing dependencies for s3tests==0.0.1 2023-01-24T17:29:14.986 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:29:14.986 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2023-01-24T17:29:15.004 INFO:tasks.s3tests:Configuring boto... 2023-01-24T17:29:15.004 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-01-24T17:29:15.005 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/boto.cfg 2023-01-24T17:29:15.062 DEBUG:teuthology.orchestra.run.smithi077:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3' 2023-01-24T17:29:15.521 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2023-01-24T17:29:15.570 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2023-01-24T17:29:15.625 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2023-01-24T17:29:15.682 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2023-01-24T17:29:15.737 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2023-01-24T17:29:15.745 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2023-01-24T17:29:15.802 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2023-01-24T17:29:15.810 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2023-01-24T17:29:15.851 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2023-01-24T17:29:15.893 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2023-01-24T17:29:15.946 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2023-01-24T17:29:16.000 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2023-01-24T17:29:16.050 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2023-01-24T17:29:16.103 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2023-01-24T17:29:16.115 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2023-01-24T17:29:16.159 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2023-01-24T17:29:16.160 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2023-01-24T17:29:16.160 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2023-01-24T17:29:16.160 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2023-01-24T17:29:16.160 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2023-01-24T17:29:16.161 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2023-01-24T17:29:16.161 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2023-01-24T17:29:16.161 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2023-01-24T17:29:16.161 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2023-01-24T17:29:16.161 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2023-01-24T17:29:16.162 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2023-01-24T17:29:16.162 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2023-01-24T17:29:16.162 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2023-01-24T17:29:16.162 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2023-01-24T17:29:16.162 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2023-01-24T17:29:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2023-01-24T17:29:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2023-01-24T17:29:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2023-01-24T17:29:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-01-24T17:29:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2023-01-24T17:29:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2023-01-24T17:29:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2023-01-24T17:29:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2023-01-24T17:29:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2023-01-24T17:29:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2023-01-24T17:29:16.165 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2023-01-24T17:29:16.165 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2023-01-24T17:29:16.165 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2023-01-24T17:29:16.165 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2023-01-24T17:29:16.165 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2023-01-24T17:29:16.166 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2023-01-24T17:29:16.166 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2023-01-24T17:29:16.166 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2023-01-24T17:29:16.166 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2023-01-24T17:29:16.166 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2023-01-24T17:29:16.167 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2023-01-24T17:29:16.167 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2023-01-24T17:29:16.167 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-01-24T17:29:16.299 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2023-01-24T17:29:16.299 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2023-01-24T17:29:16.299 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2023-01-24T17:29:16.299 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2023-01-24T17:29:16.300 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2023-01-24T17:29:16.300 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2023-01-24T17:29:16.300 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2023-01-24T17:29:16.392 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2023-01-24T17:29:16.451 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2023-01-24T17:29:18.552 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2023-01-24T17:29:18.653 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2023-01-24T17:29:18.722 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2023-01-24T17:29:18.741 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2023-01-24T17:29:18.820 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2023-01-24T17:29:18.820 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2023-01-24T17:29:18.820 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2023-01-24T17:29:18.821 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2023-01-24T17:29:18.821 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2023-01-24T17:29:18.821 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2023-01-24T17:29:18.822 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2023-01-24T17:29:18.822 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2023-01-24T17:29:18.822 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2023-01-24T17:29:18.823 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2023-01-24T17:29:18.823 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2023-01-24T17:29:18.823 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2023-01-24T17:29:18.823 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2023-01-24T17:29:18.824 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_pubilc_errordoc ... SKIP 2023-01-24T17:29:18.824 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2023-01-24T17:29:18.824 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2023-01-24T17:29:18.824 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2023-01-24T17:29:18.824 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2023-01-24T17:29:18.825 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2023-01-24T17:29:18.825 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2023-01-24T17:29:18.825 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2023-01-24T17:29:18.825 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2023-01-24T17:29:18.825 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2023-01-24T17:29:18.826 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2023-01-24T17:29:18.826 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2023-01-24T17:29:18.826 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2023-01-24T17:29:18.826 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2023-01-24T17:29:18.826 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2023-01-24T17:29:18.827 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2023-01-24T17:29:18.827 INFO:teuthology.orchestra.run.smithi077.stderr:SKIP 2023-01-24T17:29:20.464 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests.functional.test_utils.test_generate ... ok 2023-01-24T17:29:20.464 INFO:teuthology.orchestra.run.smithi077.stderr:/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2023-01-24T17:29:20.464 INFO:teuthology.orchestra.run.smithi077.stderr: warnings.warn(warning, PythonDeprecationWarning) 2023-01-24T17:29:20.489 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2023-01-24T17:29:27.493 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2023-01-24T17:29:27.526 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2023-01-24T17:29:27.576 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2023-01-24T17:29:27.615 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2023-01-24T17:29:27.658 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2023-01-24T17:29:27.700 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2023-01-24T17:29:27.748 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2023-01-24T17:29:27.786 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2023-01-24T17:29:27.817 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2023-01-24T17:29:27.849 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2023-01-24T17:29:27.872 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2023-01-24T17:29:27.888 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2023-01-24T17:29:27.903 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2023-01-24T17:29:27.911 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2023-01-24T17:29:27.936 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2023-01-24T17:29:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2023-01-24T17:29:28.020 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2023-01-24T17:29:28.044 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2023-01-24T17:29:28.074 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2023-01-24T17:29:28.100 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2023-01-24T17:29:28.133 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2023-01-24T17:29:28.158 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2023-01-24T17:29:28.178 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2023-01-24T17:29:28.197 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2023-01-24T17:29:28.206 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2023-01-24T17:29:28.214 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2023-01-24T17:29:28.221 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2023-01-24T17:29:28.231 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2023-01-24T17:29:28.239 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2023-01-24T17:29:28.324 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2023-01-24T17:29:28.420 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2023-01-24T17:29:28.524 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2023-01-24T17:29:28.615 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2023-01-24T17:29:28.704 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2023-01-24T17:29:28.756 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2023-01-24T17:29:28.834 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2023-01-24T17:29:28.898 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2023-01-24T17:29:28.952 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2023-01-24T17:29:29.074 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2023-01-24T17:29:29.200 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2023-01-24T17:29:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2023-01-24T17:29:29.290 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2023-01-24T17:29:29.348 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2023-01-24T17:29:29.407 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2023-01-24T17:29:29.535 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2023-01-24T17:29:29.681 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2023-01-24T17:29:29.735 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2023-01-24T17:29:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2023-01-24T17:29:29.850 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2023-01-24T17:29:29.912 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2023-01-24T17:29:29.974 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2023-01-24T17:29:30.029 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2023-01-24T17:29:30.092 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2023-01-24T17:29:30.153 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2023-01-24T17:29:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2023-01-24T17:29:30.268 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2023-01-24T17:29:30.322 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2023-01-24T17:29:30.383 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2023-01-24T17:29:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2023-01-24T17:29:30.501 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2023-01-24T17:29:30.551 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2023-01-24T17:29:30.617 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2023-01-24T17:29:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2023-01-24T17:29:38.286 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2023-01-24T17:29:38.350 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2023-01-24T17:29:38.400 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2023-01-24T17:29:38.480 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2023-01-24T17:29:38.534 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2023-01-24T17:29:38.586 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2023-01-24T17:29:38.650 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2023-01-24T17:29:38.706 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2023-01-24T17:29:38.755 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2023-01-24T17:29:38.819 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2023-01-24T17:29:38.877 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2023-01-24T17:29:38.931 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2023-01-24T17:29:38.994 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2023-01-24T17:29:39.050 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2023-01-24T17:29:39.114 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2023-01-24T17:29:39.185 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2023-01-24T17:29:39.240 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2023-01-24T17:29:39.301 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2023-01-24T17:29:39.366 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2023-01-24T17:29:39.444 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2023-01-24T17:29:39.508 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2023-01-24T17:29:39.562 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-01-24T17:29:39.615 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-01-24T17:29:39.717 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2023-01-24T17:29:39.816 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2023-01-24T17:29:39.872 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2023-01-24T17:29:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2023-01-24T17:29:39.982 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2023-01-24T17:29:40.039 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2023-01-24T17:29:40.316 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_account_usage ... ok 2023-01-24T17:29:40.355 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_head_bucket_usage ... ok 2023-01-24T17:29:40.781 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2023-01-24T17:29:41.199 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2023-01-24T17:29:41.264 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2023-01-24T17:29:41.329 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2023-01-24T17:29:41.396 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2023-01-24T17:29:41.453 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2023-01-24T17:29:41.568 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken ... ok 2023-01-24T17:29:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_both_continuationtoken_startafter ... ok 2023-01-24T17:29:41.739 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2023-01-24T17:29:41.797 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2023-01-24T17:29:41.862 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2023-01-24T17:29:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2023-01-24T17:29:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2023-01-24T17:29:42.031 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2023-01-24T17:29:42.299 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2023-01-24T17:29:42.619 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2023-01-24T17:29:42.654 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2023-01-24T17:29:42.692 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2023-01-24T17:29:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2023-01-24T17:29:42.753 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2023-01-24T17:29:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2023-01-24T17:29:42.772 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2023-01-24T17:29:42.780 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2023-01-24T17:29:42.831 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2023-01-24T17:29:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2023-01-24T17:29:43.006 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2023-01-24T17:29:43.112 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2023-01-24T17:29:43.140 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_read_not_exist ... ok 2023-01-24T17:29:43.165 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2023-01-24T17:29:43.388 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2023-01-24T17:29:43.620 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2023-01-24T17:29:51.556 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete_key_limit ... ok 2023-01-24T17:29:59.487 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete_key_limit ... ok 2023-01-24T17:29:59.519 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2023-01-24T17:29:59.546 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2023-01-24T17:29:59.589 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2023-01-24T17:29:59.623 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2023-01-24T17:29:59.760 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2023-01-24T17:29:59.792 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2023-01-24T17:29:59.824 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2023-01-24T17:29:59.876 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2023-01-24T17:29:59.924 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2023-01-24T17:30:00.001 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2023-01-24T17:30:00.078 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2023-01-24T17:30:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2023-01-24T17:30:00.218 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2023-01-24T17:30:00.237 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2023-01-24T17:30:00.260 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2023-01-24T17:30:00.288 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2023-01-24T17:30:00.351 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2023-01-24T17:30:00.388 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2023-01-24T17:30:00.417 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2023-01-24T17:30:00.447 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2023-01-24T17:30:00.486 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2023-01-24T17:30:00.561 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2023-01-24T17:30:00.590 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2023-01-24T17:30:00.616 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2023-01-24T17:30:00.640 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2023-01-24T17:30:00.664 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2023-01-24T17:30:00.689 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2023-01-24T17:30:00.722 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2023-01-24T17:30:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2023-01-24T17:30:00.786 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2023-01-24T17:30:00.817 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2023-01-24T17:30:00.846 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2023-01-24T17:30:00.874 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2023-01-24T17:30:00.897 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2023-01-24T17:30:00.921 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2023-01-24T17:30:00.947 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2023-01-24T17:30:00.977 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2023-01-24T17:30:01.006 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2023-01-24T17:30:01.034 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2023-01-24T17:30:01.064 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2023-01-24T17:30:01.099 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2023-01-24T17:30:01.178 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2023-01-24T17:30:01.263 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2023-01-24T17:30:01.297 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2023-01-24T17:30:01.370 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2023-01-24T17:30:01.447 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2023-01-24T17:30:02.492 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2023-01-24T17:30:02.572 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2023-01-24T17:30:02.651 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2023-01-24T17:30:02.774 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2023-01-24T17:30:02.936 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2023-01-24T17:30:03.060 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2023-01-24T17:30:03.197 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2023-01-24T17:30:03.318 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2023-01-24T17:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2023-01-24T17:30:03.562 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2023-01-24T17:30:03.726 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2023-01-24T17:30:03.759 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2023-01-24T17:30:03.838 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2023-01-24T17:30:03.923 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2023-01-24T17:30:03.966 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2023-01-24T17:30:04.000 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2023-01-24T17:30:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_notexist ... ok 2023-01-24T17:30:04.086 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2023-01-24T17:30:04.125 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2023-01-24T17:30:04.160 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2023-01-24T17:30:04.193 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2023-01-24T17:30:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2023-01-24T17:30:04.261 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2023-01-24T17:30:04.295 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2023-01-24T17:30:04.413 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2023-01-24T17:30:04.505 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2023-01-24T17:30:04.537 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2023-01-24T17:30:04.568 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2023-01-24T17:30:04.605 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2023-01-24T17:30:04.645 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2023-01-24T17:30:04.689 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2023-01-24T17:30:04.755 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2023-01-24T17:30:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2023-01-24T17:30:04.825 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2023-01-24T17:30:04.835 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2023-01-24T17:30:04.846 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2023-01-24T17:30:04.858 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2023-01-24T17:30:04.868 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2023-01-24T17:30:04.898 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2023-01-24T17:30:04.960 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2023-01-24T17:30:04.986 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2023-01-24T17:30:05.051 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl ... ok 2023-01-24T17:30:05.117 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl ... ok 2023-01-24T17:30:05.149 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2023-01-24T17:30:05.212 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2023-01-24T17:30:05.323 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2023-01-24T17:30:05.384 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2023-01-24T17:30:05.445 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2023-01-24T17:30:05.526 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2023-01-24T17:30:05.600 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2023-01-24T17:30:05.726 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2023-01-24T17:30:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2023-01-24T17:30:05.885 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2023-01-24T17:30:06.012 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2023-01-24T17:30:06.137 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2023-01-24T17:30:06.231 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2023-01-24T17:30:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2023-01-24T17:30:06.319 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2023-01-24T17:30:06.447 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2023-01-24T17:30:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2023-01-24T17:30:06.698 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2023-01-24T17:30:06.820 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2023-01-24T17:30:06.945 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2023-01-24T17:30:07.092 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2023-01-24T17:30:07.204 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2023-01-24T17:30:07.318 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2023-01-24T17:30:07.446 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2023-01-24T17:30:07.560 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2023-01-24T17:30:07.602 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2023-01-24T17:30:07.713 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2023-01-24T17:30:07.797 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2023-01-24T17:30:07.879 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2023-01-24T17:30:07.962 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2023-01-24T17:30:08.004 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_not_exist ... ok 2023-01-24T17:30:08.134 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2023-01-24T17:30:08.343 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2023-01-24T17:30:08.576 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2023-01-24T17:30:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2023-01-24T17:30:08.997 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2023-01-24T17:30:09.214 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2023-01-24T17:30:09.428 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2023-01-24T17:30:09.641 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2023-01-24T17:30:09.859 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2023-01-24T17:30:10.068 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2023-01-24T17:30:10.245 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2023-01-24T17:30:10.443 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2023-01-24T17:30:10.622 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2023-01-24T17:30:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2023-01-24T17:30:10.955 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_buckets_list_ctime ... ok 2023-01-24T17:30:10.961 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2023-01-24T17:30:10.970 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2023-01-24T17:30:10.978 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2023-01-24T17:30:11.153 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2023-01-24T17:30:11.315 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2023-01-24T17:30:11.333 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2023-01-24T17:30:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2023-01-24T17:30:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2023-01-24T17:30:12.113 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2023-01-24T17:30:12.193 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2023-01-24T17:30:12.295 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2023-01-24T17:30:12.426 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2023-01-24T17:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2023-01-24T17:30:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2023-01-24T17:30:12.738 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2023-01-24T17:30:12.914 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2023-01-24T17:30:12.989 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2023-01-24T17:30:13.108 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2023-01-24T17:30:13.252 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2023-01-24T17:30:13.475 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2023-01-24T17:30:13.670 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2023-01-24T17:30:13.698 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2023-01-24T17:30:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2023-01-24T17:30:14.334 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2023-01-24T17:30:14.495 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2023-01-24T17:30:15.609 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2023-01-24T17:30:15.649 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2023-01-24T17:30:15.732 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2023-01-24T17:30:16.036 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2023-01-24T17:30:16.124 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2023-01-24T17:30:16.365 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2023-01-24T17:30:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2023-01-24T17:30:18.109 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2023-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2023-01-24T17:30:23.922 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2023-01-24T17:30:24.775 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2023-01-24T17:30:26.216 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2023-01-24T17:30:26.353 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2023-01-24T17:30:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2023-01-24T17:30:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2023-01-24T17:30:27.226 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2023-01-24T17:30:27.303 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2023-01-24T17:30:27.628 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2023-01-24T17:30:27.782 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload_owner ... ok 2023-01-24T17:30:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2023-01-24T17:30:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2023-01-24T17:30:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2023-01-24T17:30:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2023-01-24T17:30:31.236 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2023-01-24T17:30:34.282 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2023-01-24T17:30:37.324 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2023-01-24T17:30:37.455 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_set_bucket_tagging ... ok 2023-01-24T17:30:37.563 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2023-01-24T17:30:37.741 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2023-01-24T17:30:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2023-01-24T17:30:38.180 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2023-01-24T17:30:38.439 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2023-01-24T17:30:38.827 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2023-01-24T17:30:38.950 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2023-01-24T17:30:39.217 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2023-01-24T17:30:39.668 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2023-01-24T17:30:39.795 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2023-01-24T17:30:39.972 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2023-01-24T17:30:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2023-01-24T17:30:40.331 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2023-01-24T17:30:46.134 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2023-01-24T17:30:46.211 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2023-01-24T17:30:46.292 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2023-01-24T17:30:46.374 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2023-01-24T17:30:46.448 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2023-01-24T17:30:46.689 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2023-01-24T17:30:52.833 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2023-01-24T17:30:53.128 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2023-01-24T17:30:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2023-01-24T17:30:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2023-01-24T17:30:54.023 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2023-01-24T17:30:56.375 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2023-01-24T17:30:59.505 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2023-01-24T17:31:11.923 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2023-01-24T17:31:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2023-01-24T17:31:14.051 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2023-01-24T17:31:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2023-01-24T17:31:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2023-01-24T17:31:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2023-01-24T17:31:15.562 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2023-01-24T17:31:15.867 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2023-01-24T17:31:16.036 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2023-01-24T17:31:16.518 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... ok 2023-01-24T17:31:16.561 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2023-01-24T17:31:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2023-01-24T17:31:16.712 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2023-01-24T17:31:16.740 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2023-01-24T17:31:16.773 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2023-01-24T17:31:16.892 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2023-01-24T17:31:16.936 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2023-01-24T17:31:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2023-01-24T17:31:17.023 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2023-01-24T17:31:17.054 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2023-01-24T17:31:17.084 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2023-01-24T17:31:17.128 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2023-01-24T17:31:17.164 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2023-01-24T17:31:17.201 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_set_invalid_date ... ok 2023-01-24T17:31:17.201 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition ... SKIP 2023-01-24T17:31:17.201 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2023-01-24T17:31:17.201 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2023-01-24T17:31:17.201 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition ... SKIP 2023-01-24T17:31:17.202 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_multiple_transition ... SKIP 2023-01-24T17:31:17.202 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition_large_obj ... SKIP 2023-01-24T17:31:17.285 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2023-01-24T17:31:17.365 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2023-01-24T17:31:17.419 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2023-01-24T17:31:17.494 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2023-01-24T17:31:17.540 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2023-01-24T17:31:17.626 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2023-01-24T17:31:17.706 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2023-01-24T17:31:17.776 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2023-01-24T17:31:17.845 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2023-01-24T17:31:17.914 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2023-01-24T17:31:18.008 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2023-01-24T17:31:18.893 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2023-01-24T17:31:19.483 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2023-01-24T17:31:19.519 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2023-01-24T17:31:19.573 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2023-01-24T17:31:19.648 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2023-01-24T17:31:19.719 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2023-01-24T17:31:19.746 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2023-01-24T17:31:20.638 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2023-01-24T17:31:21.153 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2023-01-24T17:31:21.651 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2023-01-24T17:31:21.693 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2023-01-24T17:31:21.769 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2023-01-24T17:31:21.845 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2023-01-24T17:31:21.899 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2023-01-24T17:31:21.984 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2023-01-24T17:31:22.068 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2023-01-24T17:31:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2023-01-24T17:31:22.166 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2023-01-24T17:31:22.227 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2023-01-24T17:31:22.296 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2023-01-24T17:31:22.462 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2023-01-24T17:31:22.597 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2023-01-24T17:31:22.772 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2023-01-24T17:31:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2023-01-24T17:31:23.071 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2023-01-24T17:31:23.237 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2023-01-24T17:31:23.410 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2023-01-24T17:31:23.586 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2023-01-24T17:31:23.758 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2023-01-24T17:31:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2023-01-24T17:31:24.192 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2023-01-24T17:31:24.306 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2023-01-24T17:31:24.347 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2023-01-24T17:31:24.474 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2023-01-24T17:31:24.622 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2023-01-24T17:31:24.803 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2023-01-24T17:31:24.988 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2023-01-24T17:31:25.249 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2023-01-24T17:31:26.966 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2023-01-24T17:31:27.118 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2023-01-24T17:31:27.328 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2023-01-24T17:31:27.517 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2023-01-24T17:31:27.795 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2023-01-24T17:31:27.968 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2023-01-24T17:31:28.062 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2023-01-24T17:31:28.241 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2023-01-24T17:31:28.270 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_s3 ... ok 2023-01-24T17:31:28.298 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_kms ... ok 2023-01-24T17:31:28.325 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_s3_kms ... ok 2023-01-24T17:31:28.356 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_bad_enc_kms ... ok 2023-01-24T17:31:28.439 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_noenc ... ok 2023-01-24T17:31:28.516 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_s3 ... ok 2023-01-24T17:31:28.777 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2023-01-24T17:31:28.852 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2023-01-24T17:31:28.915 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2023-01-24T17:31:28.981 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2023-01-24T17:31:29.046 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2023-01-24T17:31:29.110 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2023-01-24T17:31:29.222 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_mode ... ok 2023-01-24T17:31:29.288 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2023-01-24T17:31:29.356 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2023-01-24T17:31:29.432 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2023-01-24T17:31:29.502 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2023-01-24T17:31:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2023-01-24T17:31:29.640 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2023-01-24T17:31:29.770 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2023-01-24T17:31:29.930 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2023-01-24T17:31:30.023 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_iso8601 ... ok 2023-01-24T17:31:30.120 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2023-01-24T17:31:30.270 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2023-01-24T17:31:30.418 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2023-01-24T17:31:30.563 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2023-01-24T17:31:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2023-01-24T17:31:30.829 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2023-01-24T17:31:30.942 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2023-01-24T17:31:31.114 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention_and_marker ... ok 2023-01-24T17:31:31.274 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_multi_delete_object_with_retention ... ok 2023-01-24T17:31:31.320 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2023-01-24T17:31:31.396 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2023-01-24T17:31:31.477 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2023-01-24T17:31:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2023-01-24T17:31:31.695 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2023-01-24T17:31:31.793 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2023-01-24T17:31:31.855 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2023-01-24T17:31:31.942 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2023-01-24T17:31:32.009 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2023-01-24T17:31:32.064 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_with_bypass ... ok 2023-01-24T17:31:32.146 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_without_bypass ... ok 2023-01-24T17:31:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_compliance ... ok 2023-01-24T17:31:32.361 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifmatch_good ... ok 2023-01-24T17:31:32.493 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifnonematch_failed ... ok 2023-01-24T17:31:32.514 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_user_policy ... ok 2023-01-24T17:31:32.548 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2023-01-24T17:31:32.655 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2023-01-24T17:31:32.736 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2023-01-24T17:31:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2023-01-24T17:31:32.901 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2023-01-24T17:31:32.981 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2023-01-24T17:31:33.010 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2023-01-24T17:31:33.084 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2023-01-24T17:31:33.294 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2023-01-24T17:31:33.454 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2023-01-24T17:31:33.530 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2023-01-24T17:31:33.719 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2023-01-24T17:31:34.546 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_on_a_bucket_with_policy ... ok 2023-01-24T17:31:34.578 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_put_bucket_encryption_kms ... ok 2023-01-24T17:31:34.661 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_encryption_kms ... ok 2023-01-24T17:31:34.745 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_delete_bucket_encryption_kms ... ok 2023-01-24T17:31:34.829 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_post_object_authenticated_request ... ok 2023-01-24T17:31:35.766 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_generate_where_clause ... ok 2023-01-24T17:31:36.663 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_generate_projection ... ok 2023-01-24T17:31:36.716 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_count_operation ... ok 2023-01-24T17:31:37.121 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_column_sum_min_max ... ok 2023-01-24T17:31:37.674 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_nullif_expressions ... ok 2023-01-24T17:31:37.943 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_nulliftrue_expressions ... ok 2023-01-24T17:31:38.155 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_is_not_null_expressions ... ok 2023-01-24T17:31:38.203 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_lowerupper_expressions ... ok 2023-01-24T17:31:38.747 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_in_expressions ... ok 2023-01-24T17:31:39.105 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_true_false_in_expressions ... ok 2023-01-24T17:31:39.615 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_like_expressions ... ok 2023-01-24T17:31:39.956 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_truefalselike_expressions ... ok 2023-01-24T17:31:40.202 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_complex_expressions ... ok 2023-01-24T17:31:40.392 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_alias ... ok 2023-01-24T17:31:40.554 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_alias_cyclic_refernce ... ok 2023-01-24T17:31:40.758 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_datetime ... ok 2023-01-24T17:31:41.024 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_true_false_datetime ... ok 2023-01-24T17:31:41.115 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_csv_parser ... ok 2023-01-24T17:31:41.329 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_csv_definition ... ok 2023-01-24T17:31:41.539 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_schema_definition ... ok 2023-01-24T17:31:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_when_then_else_expressions ... ok 2023-01-24T17:31:42.026 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_coalesce_expressions ... ok 2023-01-24T17:31:42.239 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_cast_expressions ... ok 2023-01-24T17:31:42.240 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_version ... ok 2023-01-24T17:31:42.605 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_trim_expressions ... ok 2023-01-24T17:31:42.987 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_truefalse_trim_expressions ... ok 2023-01-24T17:31:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_escape_expressions ... ok 2023-01-24T17:31:43.432 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_case_value_expressions ... ok 2023-01-24T17:31:43.612 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_bool_cast_expressions ... ok 2023-01-24T17:31:56.591 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_progress_expressions ... ok 2023-01-24T17:31:56.592 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_s3select.test_output_serial_expressions ... ok 2023-01-24T17:33:05.695 INFO:teuthology.orchestra.run.smithi077.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2023-01-24T17:33:05.696 INFO:teuthology.orchestra.run.smithi077.stderr: 2023-01-24T17:33:05.696 INFO:teuthology.orchestra.run.smithi077.stderr:---------------------------------------------------------------------- 2023-01-24T17:33:05.696 INFO:teuthology.orchestra.run.smithi077.stderr:Ran 619 tests in 230.422s 2023-01-24T17:33:05.696 INFO:teuthology.orchestra.run.smithi077.stderr: 2023-01-24T17:33:05.696 INFO:teuthology.orchestra.run.smithi077.stderr:OK (SKIP=79) 2023-01-24T17:33:05.761 INFO:teuthology.orchestra.run.smithi077.stderr:Exception ignored in: .generate at 0x7f97bd6a70a0> 2023-01-24T17:33:05.761 INFO:teuthology.orchestra.run.smithi077.stderr:RuntimeError: generator ignored GeneratorExit 2023-01-24T17:33:05.772 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2023-01-24T17:33:05.772 DEBUG:teuthology.orchestra.run.smithi077:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2023-01-24T17:33:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T17:33:05.843 INFO:tasks.s3tests:Cleaning up boto... 2023-01-24T17:33:05.843 DEBUG:teuthology.orchestra.run.smithi077:> rm /home/ubuntu/cephtest/boto.cfg 2023-01-24T17:33:05.860 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2023-01-24T17:33:05.942 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:33:05.942 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:33:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.946+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.950 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.947+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:05.950 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.947+0000 7f9d6dad6680 20 realm 2023-01-24T17:33:05.950 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.947+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.950 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.947+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:05.951 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.947+0000 7f9d6dad6680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:33:05.951 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.947+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.951 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.948+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:05.951 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.948+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.952 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.948+0000 7f9d6dad6680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:05.952 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.948+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.952 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.949+0000 7f9d6dad6680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:33:05.953 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.949+0000 7f9d6dad6680 20 searching for the correct realm 2023-01-24T17:33:05.958 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.955+0000 7f9d6dad6680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:33:05.958 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.955+0000 7f9d6dad6680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:33:05.959 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.955+0000 7f9d6dad6680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:33:05.959 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.955+0000 7f9d6dad6680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:33:05.959 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.955+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.959 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.956+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:05.959 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.956+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.960 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.956+0000 7f9d6dad6680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:05.960 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.956+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.961 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.956+0000 7f9d6dad6680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:33:05.961 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.956+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.961 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.956+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:05.961 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.956+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.962 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.957+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:05.962 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.957+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.962 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.957+0000 7f9d6dad6680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:05.966 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.963+0000 7f9d6dad6680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:33:05.966 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.963+0000 7f9d6dad6680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:33:05.966 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.963+0000 7f9d6dad6680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:33:05.966 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.963+0000 7f9d6dad6680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:33:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.963+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.963+0000 7f9d6dad6680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.963+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.963+0000 7f9d6dad6680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:33:05.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.963+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:05.968 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.969 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:33:05.969 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.969 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:05.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 20 zone default found 2023-01-24T17:33:05.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 4 Realm: () 2023-01-24T17:33:05.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 4 ZoneGroup: default (0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f) 2023-01-24T17:33:05.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 4 Zone: default (97c5572f-f788-4cd6-8a34-3994e797b864) 2023-01-24T17:33:05.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:33:05.971 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 20 zonegroup default 2023-01-24T17:33:05.971 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.971 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:05.971 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.971 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.964+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.965+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.965+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.965+0000 7f9d6dad6680 20 started sync module instance, tier type = 2023-01-24T17:33:05.972 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.965+0000 7f9d6dad6680 20 started zone id=97c5572f-f788-4cd6-8a34-3994e797b864 (name=default) with tier type = 2023-01-24T17:33:05.978 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.975+0000 7f9d6dad6680 20 add_watcher() i=0 2023-01-24T17:33:05.978 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.975+0000 7f9d6dad6680 20 add_watcher() i=1 2023-01-24T17:33:05.978 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.975+0000 7f9d6dad6680 20 add_watcher() i=2 2023-01-24T17:33:05.978 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.975+0000 7f9d6dad6680 20 add_watcher() i=3 2023-01-24T17:33:05.978 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.975+0000 7f9d6dad6680 20 add_watcher() i=4 2023-01-24T17:33:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.975+0000 7f9d6dad6680 20 add_watcher() i=5 2023-01-24T17:33:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.975+0000 7f9d6dad6680 20 add_watcher() i=6 2023-01-24T17:33:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.976+0000 7f9d6dad6680 20 add_watcher() i=7 2023-01-24T17:33:05.979 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.976+0000 7f9d6dad6680 2 all 8 watchers are set, enabling cache 2023-01-24T17:33:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.977+0000 7f9d31fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:33:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.977+0000 7f9d6dad6680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:33:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:05.977+0000 7f9d6dad6680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:33:06.044 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.040+0000 7f9ca8ff9700 20 reqs_thread_entry: start 2023-01-24T17:33:06.044 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9c9e7fc700 20 reqs_thread_entry: start 2023-01-24T17:33:06.045 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9d6dad6680 20 init_complete bucket index max shards: 11 2023-01-24T17:33:06.045 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9d6dad6680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:33:06.045 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9c9cff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:33:36 2023 2023-01-24T17:33:06.045 INFO:teuthology.orchestra.run.smithi077.stderr: 2023-01-24T17:33:06.045 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9c83fff700 20 reqs_thread_entry: start 2023-01-24T17:33:06.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9d6dad6680 20 get_system_obj_state: rctx=0x7ffe24076598 obj=default.rgw.meta:users.uid:foo.client.0 state=0x557cc2806940 s->prefetch_data=0 2023-01-24T17:33:06.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9d6dad6680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-01-24T17:33:06.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9d6dad6680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-01-24T17:33:06.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9d6dad6680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:33:06.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9d6dad6680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.047 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9d6dad6680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:33:06.047 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9d6dad6680 20 Read xattr: user.rgw.idtag 2023-01-24T17:33:06.047 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9d6dad6680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:33:06.047 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.041+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.051 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.042+0000 7f9d6dad6680 20 rados_obj.operate() r=0 bl.length=469 2023-01-24T17:33:06.051 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.042+0000 7f9d6dad6680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-01-24T17:33:06.051 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.042+0000 7f9d6dad6680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:33:06.052 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.042+0000 7f9d6dad6680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.052 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 20 get_system_obj_state: rctx=0x7ffe24076250 obj=default.rgw.meta:root:.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 state=0x557cc2806940 s->prefetch_data=0 2023-01-24T17:33:06.052 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 : miss 2023-01-24T17:33:06.052 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 info.flags=0x16 2023-01-24T17:33:06.052 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 10 adding default.rgw.meta+root+.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 to cache LRU end 2023-01-24T17:33:06.053 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.053 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:33:06.053 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:33:06.053 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 20 Read xattr: user.rgw.acl 2023-01-24T17:33:06.053 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 20 Read xattr: user.rgw.idtag 2023-01-24T17:33:06.054 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:33:06.054 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.054 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 20 rados_obj.operate() r=0 bl.length=364 2023-01-24T17:33:06.054 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 info.flags=0x13 2023-01-24T17:33:06.054 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 10 moving default.rgw.meta+root+.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 to cache LRU end 2023-01-24T17:33:06.054 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:33:06.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 2023-01-24T17:33:06.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.043+0000 7f9d6dad6680 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :test[97c5572f-f788-4cd6-8a34-3994e797b864.4204.518]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-01-24T17:33:06.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.044+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 138BB8 2023-01-24T17:33:06.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.044+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 5A8783 2023-01-24T17:33:06.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.044+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got EB0E22 2023-01-24T17:33:06.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.044+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got FCA6BA 2023-01-24T17:33:06.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.044+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 3260E5 2023-01-24T17:33:06.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.044+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 36E738 2023-01-24T17:33:06.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.044+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got BC96DC 2023-01-24T17:33:06.057 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.044+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got F2F737 2023-01-24T17:33:06.057 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.044+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got FCCEA3 2023-01-24T17:33:06.057 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.044+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 77CC60 2023-01-24T17:33:06.057 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.044+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got B23679 2023-01-24T17:33:06.057 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.045+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got B2D703 2023-01-24T17:33:06.058 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.045+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got F3F76F 2023-01-24T17:33:06.058 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.045+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 434239 2023-01-24T17:33:06.058 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.045+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got D0FCB3 2023-01-24T17:33:06.058 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.045+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got EA6E50 2023-01-24T17:33:06.058 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.046+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 297C86 2023-01-24T17:33:06.058 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.046+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 3B2609 2023-01-24T17:33:06.059 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.046+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got E13276 2023-01-24T17:33:06.059 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.046+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 105294 2023-01-24T17:33:06.059 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.046+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 299843 2023-01-24T17:33:06.059 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.046+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got B66349 2023-01-24T17:33:06.059 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.046+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got E73011 2023-01-24T17:33:06.060 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.046+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 41C8F8 2023-01-24T17:33:06.060 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.047+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 07CE36 2023-01-24T17:33:06.060 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.047+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 34FA81 2023-01-24T17:33:06.060 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.047+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got BA89C9 2023-01-24T17:33:06.060 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.047+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 5C07AA 2023-01-24T17:33:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.047+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 83BFD8 2023-01-24T17:33:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.047+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:138BB8 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.047+0000 7f9d6dad6680 10 manifest: total_size = 398991 2023-01-24T17:33:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.047+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.14522838734081957516 2023-01-24T17:33:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.047+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:138BB8 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.061 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.047+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.047+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.0 2023-01-24T17:33:06.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.051+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : miss 2023-01-24T17:33:06.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.051+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 10 cache put: name=default.rgw.log++bucket.sync-source-hints.test info.flags=0x0 2023-01-24T17:33:06.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 10 adding default.rgw.log++bucket.sync-source-hints.test to cache LRU end 2023-01-24T17:33:06.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : miss 2023-01-24T17:33:06.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 10 cache put: name=default.rgw.log++bucket.sync-target-hints.test info.flags=0x0 2023-01-24T17:33:06.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 10 adding default.rgw.log++bucket.sync-target-hints.test to cache LRU end 2023-01-24T17:33:06.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:5A8783 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 10 manifest: total_size = 399304 2023-01-24T17:33:06.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.8143643604970201798 2023-01-24T17:33:06.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:5A8783 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.052+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.0 2023-01-24T17:33:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.060+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.060+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.060+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.060+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.060+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.060+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:EB0E22 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.060+0000 7f9d6dad6680 10 manifest: total_size = 399171 2023-01-24T17:33:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.060+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.12247741264441915885 2023-01-24T17:33:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.060+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:EB0E22 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.060+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.060+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.0 2023-01-24T17:33:06.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.066+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.066+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.066+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.066+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.066+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.066+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:FCA6BA state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.066+0000 7f9d6dad6680 10 manifest: total_size = 644584 2023-01-24T17:33:06.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.066+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.15191473007516574718 2023-01-24T17:33:06.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.066+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:FCA6BA state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.066+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.066+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.0 2023-01-24T17:33:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.073+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.073+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.073+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.073+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.073+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.073+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:3260E5 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.077 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.073+0000 7f9d6dad6680 10 manifest: total_size = 5 2023-01-24T17:33:06.077 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.073+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.9227717098144239035 2023-01-24T17:33:06.078 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.073+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:3260E5 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.078 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.073+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.078 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.073+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.1 2023-01-24T17:33:06.083 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.080+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.083 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.080+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.083 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.080+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.083 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.080+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.084 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.080+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.084 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.080+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:36E738 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.084 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.080+0000 7f9d6dad6680 10 manifest: total_size = 399146 2023-01-24T17:33:06.084 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.080+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.11114875328307483965 2023-01-24T17:33:06.084 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.080+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:36E738 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.080+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.080+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.1 2023-01-24T17:33:06.091 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.088+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.091 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.088+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.091 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.088+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.091 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.088+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.088+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.088+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:BC96DC state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.088+0000 7f9d6dad6680 10 manifest: total_size = 399114 2023-01-24T17:33:06.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.088+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.5255825143625998687 2023-01-24T17:33:06.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.088+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:BC96DC state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.088+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.088+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.1 2023-01-24T17:33:06.097 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.094+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.097 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.094+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.097 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.094+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.097 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.094+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.098 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.094+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.098 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.094+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:F2F737 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.098 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.094+0000 7f9d6dad6680 10 manifest: total_size = 399170 2023-01-24T17:33:06.098 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.094+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.12459215035818749739 2023-01-24T17:33:06.098 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.094+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:F2F737 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.099 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.094+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.099 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.094+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.1 2023-01-24T17:33:06.104 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.101+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.105 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.101+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.105 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.101+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.105 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.101+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.105 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.101+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.105 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.101+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:FCCEA3 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.106 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.102+0000 7f9d6dad6680 10 manifest: total_size = 398945 2023-01-24T17:33:06.106 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.102+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.13564509632398022394 2023-01-24T17:33:06.106 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.102+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:FCCEA3 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.106 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.102+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.106 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.102+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.1 2023-01-24T17:33:06.110 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.108+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.108+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.108+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.108+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.108+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.112 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.108+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:77CC60 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.112 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.108+0000 7f9d6dad6680 10 manifest: total_size = 399185 2023-01-24T17:33:06.112 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.108+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.9655682569914239424 2023-01-24T17:33:06.112 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.108+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:77CC60 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.112 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.108+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.113 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.108+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.2 2023-01-24T17:33:06.113 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.110+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.110+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.110+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.110+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.110+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.110+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:B23679 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.110+0000 7f9d6dad6680 10 manifest: total_size = 39909389 2023-01-24T17:33:06.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.110+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.14547250572185457477 2023-01-24T17:33:06.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.110+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:B23679 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.110+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.110+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.2 2023-01-24T17:33:06.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:33:06.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=0 2023-01-24T17:33:06.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:33:06.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=0 2023-01-24T17:33:06.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:33:06.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): result: ofs=16777216 stripe_ofs=16777216 part_ofs=0 rule->part_size=0 2023-01-24T17:33:06.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:33:06.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): result: ofs=20971520 stripe_ofs=20971520 part_ofs=0 rule->part_size=0 2023-01-24T17:33:06.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:33:06.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): result: ofs=25165824 stripe_ofs=25165824 part_ofs=0 rule->part_size=0 2023-01-24T17:33:06.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:33:06.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): result: ofs=29360128 stripe_ofs=29360128 part_ofs=0 rule->part_size=0 2023-01-24T17:33:06.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:33:06.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): result: ofs=33554432 stripe_ofs=33554432 part_ofs=0 rule->part_size=0 2023-01-24T17:33:06.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:33:06.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): result: ofs=37748736 stripe_ofs=37748736 part_ofs=0 rule->part_size=0 2023-01-24T17:33:06.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:33:06.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 RGWObjManifest::operator++(): result: ofs=39909389 stripe_ofs=39909389 part_ofs=0 rule->part_size=0 2023-01-24T17:33:06.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.113+0000 7f9d6dad6680 20 garbage collection: RGWGC::send_chain - on object name: gc.17tag is: 97c5572f-f788-4cd6-8a34-3994e797b864.4202.14547250572185457477 2023-01-24T17:33:06.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.115+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:B2D703 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.115+0000 7f9d6dad6680 10 manifest: total_size = 49277 2023-01-24T17:33:06.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.115+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.11861568064912753631 2023-01-24T17:33:06.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.115+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:B2D703 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.115+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.115+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.4 2023-01-24T17:33:06.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.117+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.117+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.117+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.117+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.117+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.117+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:F3F76F state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.117+0000 7f9d6dad6680 10 manifest: total_size = 399242 2023-01-24T17:33:06.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.117+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.7424131712880202430 2023-01-24T17:33:06.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.117+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:F3F76F state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.117+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.117+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.4 2023-01-24T17:33:06.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.120+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.120+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.120+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.120+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.120+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.120+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:434239 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.120+0000 7f9d6dad6680 10 manifest: total_size = 574494 2023-01-24T17:33:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.120+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.11261996714555705675 2023-01-24T17:33:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.120+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:434239 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.120+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.120+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.5 2023-01-24T17:33:06.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.122+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.122+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.122+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.122+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.122+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.122+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:D0FCB3 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.123+0000 7f9d6dad6680 10 manifest: total_size = 117 2023-01-24T17:33:06.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.123+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.11953035888135355858 2023-01-24T17:33:06.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.123+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:D0FCB3 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.123+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.123+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.5 2023-01-24T17:33:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.125+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.125+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.125+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.125+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.125+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.125+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:EA6E50 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.125+0000 7f9d6dad6680 10 manifest: total_size = 180000 2023-01-24T17:33:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.125+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.103730944170176945 2023-01-24T17:33:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.125+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:EA6E50 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.125+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.125+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.5 2023-01-24T17:33:06.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.128+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.128+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.128+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.128+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.128+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.128+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:297C86 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.128+0000 7f9d6dad6680 10 manifest: total_size = 398979 2023-01-24T17:33:06.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.128+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.4871651711551307652 2023-01-24T17:33:06.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.128+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:297C86 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.128+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.128+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.6 2023-01-24T17:33:06.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.131+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.131+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.131+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.131+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.131+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.131+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:3B2609 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.131+0000 7f9d6dad6680 10 manifest: total_size = 399095 2023-01-24T17:33:06.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.131+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.2972257751808766018 2023-01-24T17:33:06.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.131+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:3B2609 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.139 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.131+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.139 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.131+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.6 2023-01-24T17:33:06.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.136+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.136+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.136+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.136+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.136+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.136+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:E13276 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.136+0000 7f9d6dad6680 10 manifest: total_size = 399267 2023-01-24T17:33:06.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.136+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.17364020587368158484 2023-01-24T17:33:06.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.136+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:E13276 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.136+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.142 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.136+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.6 2023-01-24T17:33:06.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.144+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.144+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.144+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.144+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.144+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.144+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:105294 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.145+0000 7f9d6dad6680 10 manifest: total_size = 180000 2023-01-24T17:33:06.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.145+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.3258931562320619985 2023-01-24T17:33:06.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.145+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:105294 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.145+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.145+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.7 2023-01-24T17:33:06.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.153+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.153+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.153+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.153+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.153+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.153+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:299843 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.153+0000 7f9d6dad6680 10 manifest: total_size = 39 2023-01-24T17:33:06.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.153+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.7757446209553452399 2023-01-24T17:33:06.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.153+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:299843 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.153+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.153+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.7 2023-01-24T17:33:06.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.161+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.161+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.161+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.161+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.165 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.161+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.165 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.161+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:B66349 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.165 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.161+0000 7f9d6dad6680 10 manifest: total_size = 5 2023-01-24T17:33:06.165 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.161+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.17821272179952639927 2023-01-24T17:33:06.165 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.161+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:B66349 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.161+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.161+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.7 2023-01-24T17:33:06.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.167+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.167+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.167+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.167+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.171 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.167+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.171 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.167+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:E73011 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.171 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.168+0000 7f9d6dad6680 10 manifest: total_size = 360725 2023-01-24T17:33:06.172 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.168+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.6092565271491639689 2023-01-24T17:33:06.172 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.168+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:E73011 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.172 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.168+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.172 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.168+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.7 2023-01-24T17:33:06.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.175+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.175+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.175+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.175+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.175+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.175+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:41C8F8 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.175+0000 7f9d6dad6680 10 manifest: total_size = 399147 2023-01-24T17:33:06.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.175+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.10853941926381949710 2023-01-24T17:33:06.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.175+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:41C8F8 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.175+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.180 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.175+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.8 2023-01-24T17:33:06.183 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.180+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.183 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.180+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.180+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.180+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.180+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.180+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:07CE36 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.181+0000 7f9d6dad6680 10 manifest: total_size = 399016 2023-01-24T17:33:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.181+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.9577734032996775124 2023-01-24T17:33:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.181+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:07CE36 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.181+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.181+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.9 2023-01-24T17:33:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.183+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.183+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.183+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.183+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.183+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.183+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:34FA81 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.183+0000 7f9d6dad6680 10 manifest: total_size = 399108 2023-01-24T17:33:06.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.183+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.11235204039337333745 2023-01-24T17:33:06.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.183+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:34FA81 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.183+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.183+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.9 2023-01-24T17:33:06.192 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.189+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.192 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.189+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.189+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.189+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.189+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.189+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:BA89C9 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.190+0000 7f9d6dad6680 10 manifest: total_size = 589702 2023-01-24T17:33:06.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.190+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.6836158119110324878 2023-01-24T17:33:06.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.190+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:BA89C9 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.190+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.190+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.9 2023-01-24T17:33:06.200 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.197+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.200 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.197+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.197+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.197+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.197+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.197+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:5C07AA state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.197+0000 7f9d6dad6680 10 manifest: total_size = 643000 2023-01-24T17:33:06.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.197+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.17042567019167609969 2023-01-24T17:33:06.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.197+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:5C07AA state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.197+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.197+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.10 2023-01-24T17:33:06.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.205+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.205+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.205+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.205+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.205+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.205+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:83BFD8 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.205+0000 7f9d6dad6680 10 manifest: total_size = 576886 2023-01-24T17:33:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.205+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.5357827122296344972 2023-01-24T17:33:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.205+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=test:83BFD8 state=0x557cc2a9c188 s->prefetch_data=0 2023-01-24T17:33:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.205+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a9c188 2023-01-24T17:33:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.205+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.518.10 2023-01-24T17:33:06.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.213+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:33:06.216 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.213+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:33:06.216 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.213+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.216 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.213+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.216 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.213+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.216 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.213+0000 7f9d6dad6680 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-01-24T17:33:06.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.213+0000 7f9d6dad6680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :test[97c5572f-f788-4cd6-8a34-3994e797b864.4204.518]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-01-24T17:33:06.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.213+0000 7f9d6dad6680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-01-24T17:33:06.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-01-24T17:33:06.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-01-24T17:33:06.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-01-24T17:33:06.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-01-24T17:33:06.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test[97c5572f-f788-4cd6-8a34-3994e797b864.4204.518])"; uploads.size()=0; is_truncated=0 2023-01-24T17:33:06.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:test[97c5572f-f788-4cd6-8a34-3994e797b864.4204.518]) 2023-01-24T17:33:06.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 20 lifecycle: get_system_obj_state: rctx=0x7ffe24075d28 obj=default.rgw.meta:root:test state=0x557cc2806940 s->prefetch_data=0 2023-01-24T17:33:06.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 10 lifecycle: cache get: name=default.rgw.meta+root+test : miss 2023-01-24T17:33:06.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-01-24T17:33:06.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 10 lifecycle: adding default.rgw.meta+root+test to cache LRU end 2023-01-24T17:33:06.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:33:06.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 20 lifecycle: Read xattr: user.rgw.idtag 2023-01-24T17:33:06.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 10 lifecycle: cache get: name=default.rgw.meta+root+test : type miss (requested=0x13, cached=0x16) 2023-01-24T17:33:06.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 20 lifecycle: rados->read ofs=0 len=0 2023-01-24T17:33:06.222 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 20 lifecycle: rados_obj.operate() r=0 bl.length=186 2023-01-24T17:33:06.222 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x13 2023-01-24T17:33:06.222 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 10 lifecycle: moving default.rgw.meta+root+test to cache LRU end 2023-01-24T17:33:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.216+0000 7f9d6dad6680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.218+0000 7f9d6dad6680 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 info.flags=0x17 2023-01-24T17:33:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.218+0000 7f9d6dad6680 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 to cache LRU end 2023-01-24T17:33:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.218+0000 7f9d6dad6680 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:33:06.223 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.218+0000 7f9d6dad6680 10 lifecycle: distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518, ofs0, ns] 2023-01-24T17:33:06.224 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.218+0000 7f9d59ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824062 cookie 93994329120896 notifier 4244 bl.length()=800 2023-01-24T17:33:06.224 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.218+0000 7f9d59ffb700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 info.flags=0x17 2023-01-24T17:33:06.224 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.218+0000 7f9d59ffb700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 to cache LRU end 2023-01-24T17:33:06.224 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.218+0000 7f9d59ffb700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:33:06.224 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.219+0000 7f9d6dad6680 20 lifecycle: handle_bi_update(): bucket=:test[97c5572f-f788-4cd6-8a34-3994e797b864.4204.518]): orig_sources= new_sources= 2023-01-24T17:33:06.225 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.219+0000 7f9d6dad6680 20 lifecycle: handle_bi_update(): bucket=:test[97c5572f-f788-4cd6-8a34-3994e797b864.4204.518]): potential sources added=[] removed=[] 2023-01-24T17:33:06.225 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.219+0000 7f9d6dad6680 20 lifecycle: handle_bi_update(): bucket=:test[97c5572f-f788-4cd6-8a34-3994e797b864.4204.518]): orig_dests= new_dests= 2023-01-24T17:33:06.225 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.219+0000 7f9d6dad6680 20 lifecycle: handle_bi_update(): bucket=:test[97c5572f-f788-4cd6-8a34-3994e797b864.4204.518]): potential dests added=[] removed=[] 2023-01-24T17:33:06.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.226+0000 7f9d6dad6680 20 get_system_obj_state: rctx=0x7ffe24075788 obj=default.rgw.meta:root:test state=0x557cc2806940 s->prefetch_data=0 2023-01-24T17:33:06.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.226+0000 7f9d6dad6680 10 cache get: name=default.rgw.meta+root+test : type miss (requested=0x16, cached=0x13) 2023-01-24T17:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.227+0000 7f9d6dad6680 10 cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-01-24T17:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.227+0000 7f9d6dad6680 10 moving default.rgw.meta+root+test to cache LRU end 2023-01-24T17:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.227+0000 7f9d6dad6680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.227+0000 7f9d6dad6680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:33:06.231 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.227+0000 7f9d6dad6680 10 cache get: name=default.rgw.meta+root+test : hit (requested=0x11, cached=0x17) 2023-01-24T17:33:06.231 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.227+0000 7f9d6dad6680 10 removing default.rgw.meta+root+test from cache 2023-01-24T17:33:06.231 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.227+0000 7f9d6dad6680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:test, ofs0, ns] 2023-01-24T17:33:06.231 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.227+0000 7f9d5a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824432 cookie 93994329114560 notifier 4244 bl.length()=148 2023-01-24T17:33:06.232 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.228+0000 7f9d6dad6680 10 removing default.rgw.meta+root+.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 from cache 2023-01-24T17:33:06.232 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.228+0000 7f9d6dad6680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test:97c5572f-f788-4cd6-8a34-3994e797b864.4204.518, ofs0, ns] 2023-01-24T17:33:06.232 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.229+0000 7f9d59ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824063 cookie 93994329120896 notifier 4244 bl.length()=207 2023-01-24T17:33:06.241 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.238+0000 7f9d6dad6680 10 removing default.rgw.log++pubsub..bucket.test/97c5572f-f788-4cd6-8a34-3994e797b864.4204.518 from cache 2023-01-24T17:33:06.241 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.238+0000 7f9d6dad6680 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test/97c5572f-f788-4cd6-8a34-3994e797b864.4204.518, ofs0, ns] 2023-01-24T17:33:06.241 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.239+0000 7f9d5a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824433 cookie 93994329115824 notifier 4244 bl.length()=204 2023-01-24T17:33:06.244 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.241+0000 7f9d6dad6680 20 get_system_obj_state: rctx=0x7ffe24076250 obj=default.rgw.meta:root:.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 state=0x557cc2806940 s->prefetch_data=0 2023-01-24T17:33:06.244 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.241+0000 7f9d6dad6680 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 : miss 2023-01-24T17:33:06.245 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 info.flags=0x16 2023-01-24T17:33:06.245 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 10 adding default.rgw.meta+root+.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 to cache LRU end 2023-01-24T17:33:06.245 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.246 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:33:06.246 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:33:06.246 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 20 Read xattr: user.rgw.acl 2023-01-24T17:33:06.246 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 20 Read xattr: user.rgw.idtag 2023-01-24T17:33:06.246 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:33:06.246 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.247 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 20 rados_obj.operate() r=0 bl.length=369 2023-01-24T17:33:06.247 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 info.flags=0x13 2023-01-24T17:33:06.247 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 10 moving default.rgw.meta+root+.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 to cache LRU end 2023-01-24T17:33:06.247 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.247 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:33:06.247 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 2023-01-24T17:33:06.248 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.242+0000 7f9d6dad6680 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :testbuck2[97c5572f-f788-4cd6-8a34-3994e797b864.4204.519]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-01-24T17:33:06.248 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.243+0000 7f9d6dad6680 10 cls_bucket_list_unordered: got 4C9F88 2023-01-24T17:33:06.249 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.245+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=testbuck2:4C9F88 state=0x557cc2a99888 s->prefetch_data=0 2023-01-24T17:33:06.249 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.246+0000 7f9d6dad6680 10 manifest: total_size = 399170 2023-01-24T17:33:06.249 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.246+0000 7f9d6dad6680 20 get_obj_state: setting s->obj_tag to 97c5572f-f788-4cd6-8a34-3994e797b864.4202.68959705615796254 2023-01-24T17:33:06.249 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.246+0000 7f9d6dad6680 20 get_obj_state: rctx=0x7ffe24076260 obj=testbuck2:4C9F88 state=0x557cc2a99888 s->prefetch_data=0 2023-01-24T17:33:06.249 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.246+0000 7f9d6dad6680 20 prepare_atomic_modification: state is not atomic. state=0x557cc2a99888 2023-01-24T17:33:06.250 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.246+0000 7f9d6dad6680 20 bucket index object: default.rgw.buckets.index:.dir.97c5572f-f788-4cd6-8a34-3994e797b864.4204.519.2 2023-01-24T17:33:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.testbuck2 : miss 2023-01-24T17:33:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 10 cache put: name=default.rgw.log++bucket.sync-source-hints.testbuck2 info.flags=0x0 2023-01-24T17:33:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 10 adding default.rgw.log++bucket.sync-source-hints.testbuck2 to cache LRU end 2023-01-24T17:33:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.testbuck2 : miss 2023-01-24T17:33:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 10 cache put: name=default.rgw.log++bucket.sync-target-hints.testbuck2 info.flags=0x0 2023-01-24T17:33:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 10 adding default.rgw.log++bucket.sync-target-hints.testbuck2 to cache LRU end 2023-01-24T17:33:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 10 chain_cache_entry: cache_locator= 2023-01-24T17:33:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:33:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:33:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-01-24T17:33:06.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :testbuck2[97c5572f-f788-4cd6-8a34-3994e797b864.4204.519]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-01-24T17:33:06.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.248+0000 7f9d6dad6680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-01-24T17:33:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-01-24T17:33:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-01-24T17:33:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-01-24T17:33:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-01-24T17:33:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":testbuck2[97c5572f-f788-4cd6-8a34-3994e797b864.4204.519])"; uploads.size()=0; is_truncated=0 2023-01-24T17:33:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:testbuck2[97c5572f-f788-4cd6-8a34-3994e797b864.4204.519]) 2023-01-24T17:33:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 20 lifecycle: get_system_obj_state: rctx=0x7ffe24075d28 obj=default.rgw.meta:root:testbuck2 state=0x557cc2806940 s->prefetch_data=0 2023-01-24T17:33:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : miss 2023-01-24T17:33:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-01-24T17:33:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 10 lifecycle: adding default.rgw.meta+root+testbuck2 to cache LRU end 2023-01-24T17:33:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:33:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 20 lifecycle: Read xattr: user.rgw.idtag 2023-01-24T17:33:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:33:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.250+0000 7f9d6dad6680 20 lifecycle: rados->read ofs=0 len=0 2023-01-24T17:33:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.251+0000 7f9d6dad6680 20 lifecycle: rados_obj.operate() r=0 bl.length=191 2023-01-24T17:33:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.251+0000 7f9d6dad6680 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x13 2023-01-24T17:33:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.251+0000 7f9d6dad6680 10 lifecycle: moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-01-24T17:33:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.251+0000 7f9d6dad6680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.252+0000 7f9d6dad6680 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 info.flags=0x17 2023-01-24T17:33:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.252+0000 7f9d6dad6680 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 to cache LRU end 2023-01-24T17:33:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.252+0000 7f9d6dad6680 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:33:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.252+0000 7f9d6dad6680 10 lifecycle: distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519, ofs0, ns] 2023-01-24T17:33:06.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.253+0000 7f9d59ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824064 cookie 93994329120896 notifier 4244 bl.length()=810 2023-01-24T17:33:06.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.253+0000 7f9d59ffb700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 info.flags=0x17 2023-01-24T17:33:06.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.253+0000 7f9d59ffb700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 to cache LRU end 2023-01-24T17:33:06.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.253+0000 7f9d59ffb700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:33:06.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.253+0000 7f9d6dad6680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[97c5572f-f788-4cd6-8a34-3994e797b864.4204.519]): orig_sources= new_sources= 2023-01-24T17:33:06.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.253+0000 7f9d6dad6680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[97c5572f-f788-4cd6-8a34-3994e797b864.4204.519]): potential sources added=[] removed=[] 2023-01-24T17:33:06.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.253+0000 7f9d6dad6680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[97c5572f-f788-4cd6-8a34-3994e797b864.4204.519]): orig_dests= new_dests= 2023-01-24T17:33:06.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.253+0000 7f9d6dad6680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[97c5572f-f788-4cd6-8a34-3994e797b864.4204.519]): potential dests added=[] removed=[] 2023-01-24T17:33:06.266 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.263+0000 7f9d6dad6680 20 get_system_obj_state: rctx=0x7ffe24075788 obj=default.rgw.meta:root:testbuck2 state=0x557cc2806940 s->prefetch_data=0 2023-01-24T17:33:06.266 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.263+0000 7f9d6dad6680 10 cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x16, cached=0x13) 2023-01-24T17:33:06.266 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.263+0000 7f9d6dad6680 10 cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-01-24T17:33:06.267 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.263+0000 7f9d6dad6680 10 moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-01-24T17:33:06.267 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.263+0000 7f9d6dad6680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.267 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.263+0000 7f9d6dad6680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:33:06.267 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.263+0000 7f9d6dad6680 10 cache get: name=default.rgw.meta+root+testbuck2 : hit (requested=0x11, cached=0x17) 2023-01-24T17:33:06.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.263+0000 7f9d6dad6680 10 removing default.rgw.meta+root+testbuck2 from cache 2023-01-24T17:33:06.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.263+0000 7f9d6dad6680 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:testbuck2, ofs0, ns] 2023-01-24T17:33:06.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.264+0000 7f9d59ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824434 cookie 93994329115824 notifier 4244 bl.length()=153 2023-01-24T17:33:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.265+0000 7f9d6dad6680 10 removing default.rgw.meta+root+.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 from cache 2023-01-24T17:33:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.265+0000 7f9d6dad6680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.testbuck2:97c5572f-f788-4cd6-8a34-3994e797b864.4204.519, ofs0, ns] 2023-01-24T17:33:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.265+0000 7f9d5a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824065 cookie 93994329120896 notifier 4244 bl.length()=212 2023-01-24T17:33:06.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.277+0000 7f9d6dad6680 10 removing default.rgw.log++pubsub..bucket.testbuck2/97c5572f-f788-4cd6-8a34-3994e797b864.4204.519 from cache 2023-01-24T17:33:06.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.277+0000 7f9d6dad6680 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.testbuck2/97c5572f-f788-4cd6-8a34-3994e797b864.4204.519, ofs0, ns] 2023-01-24T17:33:06.280 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.277+0000 7f9d59ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824435 cookie 93994329115824 notifier 4244 bl.length()=209 2023-01-24T17:33:06.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.280+0000 7f9d6dad6680 10 removing key index: MKGTKFEOGFFSSUHDNLRG 2023-01-24T17:33:06.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.280+0000 7f9d6dad6680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.keys:MKGTKFEOGFFSSUHDNLRG, ofs0, ns] 2023-01-24T17:33:06.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.280+0000 7f9d5a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824066 cookie 93994329091184 notifier 4244 bl.length()=170 2023-01-24T17:33:06.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.282+0000 7f9d6dad6680 10 removing email index: foo.client.0+test@test.test 2023-01-24T17:33:06.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.282+0000 7f9d6dad6680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-01-24T17:33:06.285 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.282+0000 7f9d59ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824436 cookie 93994329114560 notifier 4244 bl.length()=178 2023-01-24T17:33:06.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.284+0000 7f9d6dad6680 10 removing user buckets index 2023-01-24T17:33:06.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.284+0000 7f9d6dad6680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2023-01-24T17:33:06.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.285+0000 7f9d5a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824437 cookie 93994329106256 notifier 4244 bl.length()=169 2023-01-24T17:33:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.287+0000 7f9d6dad6680 10 removing user index: foo.client.0 2023-01-24T17:33:06.289 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.287+0000 7f9d6dad6680 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2023-01-24T17:33:06.290 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.287+0000 7f9d6dad6680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-01-24T17:33:06.290 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.287+0000 7f9d59ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824067 cookie 93994329091184 notifier 4244 bl.length()=161 2023-01-24T17:33:06.292 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.289+0000 7f9d6dad6680 20 remove_watcher() i=0 2023-01-24T17:33:06.292 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.289+0000 7f9d6dad6680 2 removed watcher, disabling cache 2023-01-24T17:33:06.293 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.291+0000 7f9d6dad6680 20 remove_watcher() i=1 2023-01-24T17:33:06.296 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.294+0000 7f9d6dad6680 20 remove_watcher() i=2 2023-01-24T17:33:06.299 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.296+0000 7f9d6dad6680 20 remove_watcher() i=3 2023-01-24T17:33:06.299 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.297+0000 7f9d6dad6680 20 remove_watcher() i=4 2023-01-24T17:33:06.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.299+0000 7f9d6dad6680 20 remove_watcher() i=5 2023-01-24T17:33:06.304 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.302+0000 7f9d6dad6680 20 remove_watcher() i=6 2023-01-24T17:33:06.305 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.303+0000 7f9d6dad6680 20 remove_watcher() i=7 2023-01-24T17:33:06.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.303+0000 7f9d31fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:33:06.311 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2023-01-24T17:33:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:33:06.394 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:33:06.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.397+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.398+0000 7fa87366b680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.398+0000 7fa87366b680 20 realm 2023-01-24T17:33:06.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.398+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.398+0000 7fa87366b680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.398+0000 7fa87366b680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:33:06.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.398+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.398+0000 7fa87366b680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.398+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.399+0000 7fa87366b680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:06.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.399+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.399+0000 7fa87366b680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:33:06.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.399+0000 7fa87366b680 20 searching for the correct realm 2023-01-24T17:33:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.406+0000 7fa87366b680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:33:06.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.406+0000 7fa87366b680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:33:06.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.406+0000 7fa87366b680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:33:06.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.406+0000 7fa87366b680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:33:06.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.406+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.407+0000 7fa87366b680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.407+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.407+0000 7fa87366b680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:06.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.407+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.407+0000 7fa87366b680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:33:06.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.407+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.407+0000 7fa87366b680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.407+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.407+0000 7fa87366b680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.407+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.407+0000 7fa87366b680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:33:06.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:33:06.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:33:06.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:33:06.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:33:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:33:06.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.414+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 20 zone default found 2023-01-24T17:33:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 4 Realm: () 2023-01-24T17:33:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 4 ZoneGroup: default (0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f) 2023-01-24T17:33:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 4 Zone: default (97c5572f-f788-4cd6-8a34-3994e797b864) 2023-01-24T17:33:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:33:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 20 zonegroup default 2023-01-24T17:33:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 20 started sync module instance, tier type = 2023-01-24T17:33:06.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.415+0000 7fa87366b680 20 started zone id=97c5572f-f788-4cd6-8a34-3994e797b864 (name=default) with tier type = 2023-01-24T17:33:06.444 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.441+0000 7fa87366b680 20 add_watcher() i=0 2023-01-24T17:33:06.444 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.441+0000 7fa87366b680 20 add_watcher() i=1 2023-01-24T17:33:06.444 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.441+0000 7fa87366b680 20 add_watcher() i=2 2023-01-24T17:33:06.444 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.441+0000 7fa87366b680 20 add_watcher() i=3 2023-01-24T17:33:06.444 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.441+0000 7fa87366b680 20 add_watcher() i=4 2023-01-24T17:33:06.445 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.441+0000 7fa87366b680 20 add_watcher() i=5 2023-01-24T17:33:06.445 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.441+0000 7fa87366b680 20 add_watcher() i=6 2023-01-24T17:33:06.445 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.442+0000 7fa87366b680 20 add_watcher() i=7 2023-01-24T17:33:06.445 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.442+0000 7fa87366b680 2 all 8 watchers are set, enabling cache 2023-01-24T17:33:06.446 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.443+0000 7fa837fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:33:06.446 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.443+0000 7fa87366b680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:33:06.446 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.443+0000 7fa87366b680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:33:06.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa7a6ffd700 20 reqs_thread_entry: start 2023-01-24T17:33:06.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa7a4ff9700 20 reqs_thread_entry: start 2023-01-24T17:33:06.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa87366b680 20 init_complete bucket index max shards: 11 2023-01-24T17:33:06.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa87366b680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:33:06.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa79affd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:33:36 2023 2023-01-24T17:33:06.492 INFO:teuthology.orchestra.run.smithi077.stderr: 2023-01-24T17:33:06.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa79a7fc700 20 reqs_thread_entry: start 2023-01-24T17:33:06.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa87366b680 20 get_system_obj_state: rctx=0x7ffc7ccf13f8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x564538585940 s->prefetch_data=0 2023-01-24T17:33:06.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa87366b680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-01-24T17:33:06.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa87366b680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-01-24T17:33:06.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa87366b680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:33:06.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa87366b680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa87366b680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:33:06.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa87366b680 20 Read xattr: user.rgw.idtag 2023-01-24T17:33:06.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa87366b680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:33:06.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.488+0000 7fa87366b680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.489+0000 7fa87366b680 20 rados_obj.operate() r=0 bl.length=469 2023-01-24T17:33:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.489+0000 7fa87366b680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-01-24T17:33:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.489+0000 7fa87366b680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:33:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.489+0000 7fa87366b680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.490+0000 7fa87366b680 10 removing key index: BUQJDKRXKEPYNHRODVNF 2023-01-24T17:33:06.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.490+0000 7fa87366b680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.keys:BUQJDKRXKEPYNHRODVNF, ofs0, ns] 2023-01-24T17:33:06.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.490+0000 7fa864f50700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824068 cookie 94855299666752 notifier 4250 bl.length()=170 2023-01-24T17:33:06.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.491+0000 7fa87366b680 10 removing email index: bar.client.0+test@test.test 2023-01-24T17:33:06.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.491+0000 7fa87366b680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-01-24T17:33:06.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.492+0000 7fa864f50700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824438 cookie 94855299653200 notifier 4250 bl.length()=178 2023-01-24T17:33:06.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.493+0000 7fa87366b680 10 removing user buckets index 2023-01-24T17:33:06.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.493+0000 7fa87366b680 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2023-01-24T17:33:06.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.493+0000 7fa85f7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618822762 cookie 94855299656896 notifier 4250 bl.length()=169 2023-01-24T17:33:06.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.496+0000 7fa87366b680 10 removing user index: bar.client.0 2023-01-24T17:33:06.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.496+0000 7fa87366b680 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2023-01-24T17:33:06.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.496+0000 7fa87366b680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-01-24T17:33:06.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.496+0000 7fa85f7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824439 cookie 94855299653200 notifier 4250 bl.length()=161 2023-01-24T17:33:06.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.498+0000 7fa87366b680 20 remove_watcher() i=0 2023-01-24T17:33:06.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.498+0000 7fa87366b680 2 removed watcher, disabling cache 2023-01-24T17:33:06.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.499+0000 7fa87366b680 20 remove_watcher() i=1 2023-01-24T17:33:06.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.501+0000 7fa87366b680 20 remove_watcher() i=2 2023-01-24T17:33:06.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.502+0000 7fa87366b680 20 remove_watcher() i=3 2023-01-24T17:33:06.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.503+0000 7fa87366b680 20 remove_watcher() i=4 2023-01-24T17:33:06.507 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.504+0000 7fa87366b680 20 remove_watcher() i=5 2023-01-24T17:33:06.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.507+0000 7fa87366b680 20 remove_watcher() i=6 2023-01-24T17:33:06.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.508+0000 7fa87366b680 20 remove_watcher() i=7 2023-01-24T17:33:06.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.508+0000 7fa837fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:33:06.516 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2023-01-24T17:33:06.599 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:33:06.599 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:33:06.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.602+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.603+0000 7f7a1633e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.603+0000 7f7a1633e680 20 realm 2023-01-24T17:33:06.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.603+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.604+0000 7f7a1633e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.604+0000 7f7a1633e680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:33:06.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.604+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.604+0000 7f7a1633e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.604+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.604+0000 7f7a1633e680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:06.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.604+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.605+0000 7f7a1633e680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:33:06.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.605+0000 7f7a1633e680 20 searching for the correct realm 2023-01-24T17:33:06.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.612+0000 7f7a1633e680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:33:06.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.612+0000 7f7a1633e680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:33:06.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.612+0000 7f7a1633e680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:33:06.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.612+0000 7f7a1633e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:33:06.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.612+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.612+0000 7f7a1633e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.612+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.617 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.613+0000 7f7a1633e680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:06.617 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.613+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.618 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.613+0000 7f7a1633e680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:33:06.618 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.613+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.618 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.613+0000 7f7a1633e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.618 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.613+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.618 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.613+0000 7f7a1633e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.619 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.613+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.619 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.614+0000 7f7a1633e680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:06.623 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.620+0000 7f7a1633e680 20 RGWRados::pool_iterate: got zonegroup_info.0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f 2023-01-24T17:33:06.624 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.620+0000 7f7a1633e680 20 RGWRados::pool_iterate: got zone_info.97c5572f-f788-4cd6-8a34-3994e797b864 2023-01-24T17:33:06.624 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.620+0000 7f7a1633e680 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:33:06.624 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.620+0000 7f7a1633e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:33:06.624 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.620+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.624 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.621+0000 7f7a1633e680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:06.625 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.621+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.625 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.621+0000 7f7a1633e680 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:33:06.625 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.621+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.626 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.621+0000 7f7a1633e680 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:33:06.627 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.621+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.627 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:33:06.627 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.627 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.627 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 zone default found 2023-01-24T17:33:06.627 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 4 Realm: () 2023-01-24T17:33:06.628 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 4 ZoneGroup: default (0706c4fd-926e-4b4e-a1f8-c8dffc7f4c6f) 2023-01-24T17:33:06.628 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 4 Zone: default (97c5572f-f788-4cd6-8a34-3994e797b864) 2023-01-24T17:33:06.628 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:33:06.628 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 zonegroup default 2023-01-24T17:33:06.628 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.629 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.629 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.629 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.629 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.629 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:33:06.629 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 started sync module instance, tier type = 2023-01-24T17:33:06.630 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.622+0000 7f7a1633e680 20 started zone id=97c5572f-f788-4cd6-8a34-3994e797b864 (name=default) with tier type = 2023-01-24T17:33:06.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.646+0000 7f7a1633e680 20 add_watcher() i=0 2023-01-24T17:33:06.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.646+0000 7f7a1633e680 20 add_watcher() i=1 2023-01-24T17:33:06.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.646+0000 7f7a1633e680 20 add_watcher() i=2 2023-01-24T17:33:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.646+0000 7f7a1633e680 20 add_watcher() i=3 2023-01-24T17:33:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.646+0000 7f7a1633e680 20 add_watcher() i=4 2023-01-24T17:33:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.646+0000 7f7a1633e680 20 add_watcher() i=5 2023-01-24T17:33:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.646+0000 7f7a1633e680 20 add_watcher() i=6 2023-01-24T17:33:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.647+0000 7f7a1633e680 20 add_watcher() i=7 2023-01-24T17:33:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.647+0000 7f7a1633e680 2 all 8 watchers are set, enabling cache 2023-01-24T17:33:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.649+0000 7f79da7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:33:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.649+0000 7f7a1633e680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:33:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.649+0000 7f7a1633e680 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:33:06.702 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.697+0000 7f79497fa700 20 reqs_thread_entry: start 2023-01-24T17:33:06.702 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7942ffd700 20 reqs_thread_entry: start 2023-01-24T17:33:06.702 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 20 init_complete bucket index max shards: 11 2023-01-24T17:33:06.702 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:33:06.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f79417fa700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:33:37 2023 2023-01-24T17:33:06.703 INFO:teuthology.orchestra.run.smithi077.stderr: 2023-01-24T17:33:06.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7940ff9700 20 reqs_thread_entry: start 2023-01-24T17:33:06.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 20 get_system_obj_state: rctx=0x7ffee8b9cca8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55fd39f25940 s->prefetch_data=0 2023-01-24T17:33:06.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-01-24T17:33:06.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-01-24T17:33:06.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:33:06.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 10 updating xattr: name=user.rgw.user-policy bl.length()=118 2023-01-24T17:33:06.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:33:06.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 20 Read xattr: user.rgw.idtag 2023-01-24T17:33:06.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 20 Read xattr: user.rgw.user-policy 2023-01-24T17:33:06.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:33:06.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.698+0000 7f7a1633e680 20 rados->read ofs=0 len=0 2023-01-24T17:33:06.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.699+0000 7f7a1633e680 20 rados_obj.operate() r=0 bl.length=528 2023-01-24T17:33:06.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.699+0000 7f7a1633e680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-01-24T17:33:06.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.699+0000 7f7a1633e680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:33:06.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.699+0000 7f7a1633e680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:33:06.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.699+0000 7f7a1633e680 10 updating xattr: name=user.rgw.user-policy bl.length()=118 2023-01-24T17:33:06.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.699+0000 7f7a1633e680 10 removing key index: PKYDOPQFLRQMFLXCTRCV 2023-01-24T17:33:06.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.699+0000 7f7a1633e680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.keys:PKYDOPQFLRQMFLXCTRCV, ofs0, ns] 2023-01-24T17:33:06.707 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.699+0000 7f7a02ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824069 cookie 94546088894576 notifier 4256 bl.length()=170 2023-01-24T17:33:06.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.701+0000 7f7a1633e680 10 removing email index: testx$tenanteduser.client.0+test@test.test 2023-01-24T17:33:06.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.701+0000 7f7a1633e680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-01-24T17:33:06.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.701+0000 7f7a02ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824070 cookie 94546088894576 notifier 4256 bl.length()=193 2023-01-24T17:33:06.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.703+0000 7f7a1633e680 10 removing user buckets index 2023-01-24T17:33:06.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.703+0000 7f7a1633e680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2023-01-24T17:33:06.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.703+0000 7f7a027fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824440 cookie 94546088884576 notifier 4256 bl.length()=184 2023-01-24T17:33:06.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.705+0000 7f7a1633e680 10 removing user index: testx$tenanteduser.client.0 2023-01-24T17:33:06.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.705+0000 7f7a1633e680 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2023-01-24T17:33:06.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.705+0000 7f7a1633e680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-01-24T17:33:06.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.705+0000 7f7a02ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824071 cookie 94546088882208 notifier 4256 bl.length()=176 2023-01-24T17:33:06.711 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.709+0000 7f7a1633e680 20 remove_watcher() i=0 2023-01-24T17:33:06.711 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.709+0000 7f7a1633e680 2 removed watcher, disabling cache 2023-01-24T17:33:06.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.712+0000 7f7a1633e680 20 remove_watcher() i=1 2023-01-24T17:33:06.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.714+0000 7f7a1633e680 20 remove_watcher() i=2 2023-01-24T17:33:06.720 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.717+0000 7f7a1633e680 20 remove_watcher() i=3 2023-01-24T17:33:06.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.720+0000 7f7a1633e680 20 remove_watcher() i=4 2023-01-24T17:33:06.723 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.721+0000 7f7a1633e680 20 remove_watcher() i=5 2023-01-24T17:33:06.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.723+0000 7f7a1633e680 20 remove_watcher() i=6 2023-01-24T17:33:06.728 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.726+0000 7f7a1633e680 20 remove_watcher() i=7 2023-01-24T17:33:06.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-01-24T17:33:06.726+0000 7f79da7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:33:06.733 INFO:tasks.s3tests:Removing s3-tests... 2023-01-24T17:33:06.734 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/s3-tests 2023-01-24T17:33:06.890 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2023-01-24T17:33:06.900 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-01-24T17:33:06.908 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-01-24T17:33:06.909 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:33:06.909 INFO:tasks.rgw.client.0.smithi077.stdout:2023-01-24T17:33:06.907+0000 7fd286c51700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi077.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --rgw_crypt_sse_s3_vault_addr smithi077.front.sepia.ceph.com:8200 --rgw_crypt_sse_s3_vault_token_file /etc/ceph/vault-root-token --foreground (PID: 108663) UID: 0 2023-01-24T17:33:06.910 INFO:tasks.rgw.client.0.smithi077.stdout:2023-01-24T17:33:06.907+0000 7fd296fa6740 -1 shutting down 2023-01-24T17:33:09.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2023-01-24T17:33:13.011 INFO:tasks.rgw.client.0:Stopped 2023-01-24T17:33:13.012 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-01-24T17:33:13.036 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/vault-root-token 2023-01-24T17:33:13.110 DEBUG:teuthology.run_tasks:Unwinding manager vault 2023-01-24T17:33:13.121 INFO:tasks.vault:Stopping Vault instance 2023-01-24T17:33:13.121 DEBUG:tasks.vault.client.0:waiting for process to exit 2023-01-24T17:33:13.122 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:33:13.151 INFO:tasks.vault.client.0.smithi077.stdout:==> Vault shutdown triggered 2023-01-24T17:33:13.153 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:33:13.150Z [INFO] core: marked as sealed 2023-01-24T17:33:13.153 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:33:13.150Z [INFO] core: pre-seal teardown starting 2023-01-24T17:33:13.153 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:33:13.150Z [INFO] rollback: stopping rollback manager 2023-01-24T17:33:13.153 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:33:13.150Z [INFO] core: pre-seal teardown complete 2023-01-24T17:33:13.153 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:33:13.150Z [INFO] core: stopping cluster listeners 2023-01-24T17:33:13.154 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:33:13.150Z [INFO] core.cluster-listener: forwarding rpc listeners stopped 2023-01-24T17:33:13.314 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:33:13.313Z [INFO] core.cluster-listener: rpc listeners successfully shut down 2023-01-24T17:33:13.315 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:33:13.313Z [INFO] core: cluster listeners successfully shut down 2023-01-24T17:33:13.316 INFO:tasks.vault.client.0.smithi077.stderr:2023-01-24T17:33:13.313Z [INFO] core: vault is sealed 2023-01-24T17:33:13.317 INFO:tasks.vault.client.0:Stopped 2023-01-24T17:33:13.317 INFO:tasks.vault:Removing Vault... 2023-01-24T17:33:13.317 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/vault /home/ubuntu/cephtest/vault.zip 2023-01-24T17:33:13.395 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-01-24T17:33:13.404 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-01-24T17:33:13.404 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T17:33:13.746 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-01-24T17:33:13.747 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"fsid":"42f52ac0-10ca-44e3-9331-cf221ec3091f","created":"2023-01-24T17:27:51.417263+0000","modified":"2023-01-24T17:29:58.499798+0000","last_up_change":"2023-01-24T17:27:59.413896+0000","last_in_change":"2023-01-24T17:27:52.418578+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T17:28:00.113996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T17:28:00.952800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-01-24T17:28:17.543013+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-01-24T17:28:18.948881+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-01-24T17:28:21.071107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-01-24T17:28:22.679813+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-01-24T17:28:24.663388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-01-24T17:28:26.740922+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2023-01-24T17:28:36.846236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2023-01-24T17:29:16.477633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"9e67d55f-2f53-4427-9576-e00932ec8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":103662},{"type":"v1","addr":"172.21.15.77:6801","nonce":103662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":103662},{"type":"v1","addr":"172.21.15.77:6803","nonce":103662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":103662},{"type":"v1","addr":"172.21.15.77:6807","nonce":103662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":103662},{"type":"v1","addr":"172.21.15.77:6805","nonce":103662}]},"public_addr":"172.21.15.77:6801/103662","cluster_addr":"172.21.15.77:6803/103662","heartbeat_back_addr":"172.21.15.77:6807/103662","heartbeat_front_addr":"172.21.15.77:6805/103662","state":["exists","up"]},{"osd":1,"uuid":"292dd058-5168-46f0-8c25-8e5981e647a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":103663},{"type":"v1","addr":"172.21.15.77:6809","nonce":103663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":103663},{"type":"v1","addr":"172.21.15.77:6811","nonce":103663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":103663},{"type":"v1","addr":"172.21.15.77:6815","nonce":103663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":103663},{"type":"v1","addr":"172.21.15.77:6813","nonce":103663}]},"public_addr":"172.21.15.77:6809/103663","cluster_addr":"172.21.15.77:6811/103663","heartbeat_back_addr":"172.21.15.77:6815/103663","heartbeat_front_addr":"172.21.15.77:6813/103663","state":["exists","up"]},{"osd":2,"uuid":"3abf1aa7-dfb2-452e-b61c-f3b192ca00c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":103693},{"type":"v1","addr":"172.21.15.77:6817","nonce":103693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":103693},{"type":"v1","addr":"172.21.15.77:6819","nonce":103693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":103693},{"type":"v1","addr":"172.21.15.77:6823","nonce":103693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":103693},{"type":"v1","addr":"172.21.15.77:6821","nonce":103693}]},"public_addr":"172.21.15.77:6817/103693","cluster_addr":"172.21.15.77:6819/103693","heartbeat_back_addr":"172.21.15.77:6823/103693","heartbeat_front_addr":"172.21.15.77:6821/103693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:27:56.533062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:27:56.834326+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T17:27:56.759525+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":0,"to":2}]},{"pgid":"3.36","mappings":[{"from":0,"to":1}]},{"pgid":"4.8","mappings":[{"from":1,"to":0}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]},{"pgid":"4.21","mappings":[{"from":1,"to":2}]},{"pgid":"4.2a","mappings":[{"from":1,"to":2}]},{"pgid":"4.39","mappings":[{"from":1,"to":2}]},{"pgid":"4.3b","mappings":[{"from":1,"to":2}]},{"pgid":"4.3d","mappings":[{"from":1,"to":2}]},{"pgid":"10.13","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:33:13.760 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-01-24T17:33:14.079 INFO:teuthology.misc:Shutting down mds daemons... 2023-01-24T17:33:14.080 INFO:teuthology.misc:Shutting down osd daemons... 2023-01-24T17:33:14.080 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-01-24T17:33:14.080 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:33:14.125 INFO:tasks.ceph.osd.0:Stopped 2023-01-24T17:33:14.125 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-01-24T17:33:14.125 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:33:14.169 INFO:tasks.ceph.osd.1:Stopped 2023-01-24T17:33:14.169 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-01-24T17:33:14.170 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:33:14.215 INFO:tasks.ceph.osd.2:Stopped 2023-01-24T17:33:14.215 INFO:teuthology.misc:Shutting down mgr daemons... 2023-01-24T17:33:14.216 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-01-24T17:33:14.216 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:33:14.238 INFO:tasks.ceph.mgr.x:Stopped 2023-01-24T17:33:14.239 INFO:teuthology.misc:Shutting down mon daemons... 2023-01-24T17:33:14.239 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-01-24T17:33:14.240 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:33:14.288 INFO:tasks.ceph.mon.a:Stopped 2023-01-24T17:33:14.289 INFO:tasks.ceph:Checking cluster log for badness... 2023-01-24T17:33:14.289 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-01-24T17:33:14.357 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi077.front.sepia.ceph.com 2023-01-24T17:33:14.357 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-01-24T17:33:14.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2023-01-24T17:33:16.654 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi077.front.sepia.ceph.com 2023-01-24T17:33:16.655 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-01-24T17:33:16.761 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi077.front.sepia.ceph.com 2023-01-24T17:33:16.761 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-01-24T17:33:16.847 INFO:tasks.ceph:Archiving mon data... 2023-01-24T17:33:16.848 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135503/data/mon.a.tgz 2023-01-24T17:33:16.848 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2023-01-24T17:33:16.867 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.IUyLwmgxtQ 2023-01-24T17:33:16.868 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.IUyLwmgxtQ 2023-01-24T17:33:17.024 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.IUyLwmgxtQ 2023-01-24T17:33:17.112 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.IUyLwmgxtQ is 290KB 2023-01-24T17:33:17.206 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.IUyLwmgxtQ 2023-01-24T17:33:17.223 INFO:tasks.ceph:Cleaning ceph cluster... 2023-01-24T17:33:17.224 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-01-24T17:33:17.290 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-01-24T17:33:17.290 DEBUG:teuthology.orchestra.run.smithi077:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-01-24T17:33:17.358 INFO:teuthology.orchestra.run.smithi077.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-01-24T17:33:17.363 INFO:tasks.ceph:Archiving crash dumps... 2023-01-24T17:33:17.365 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135503/remote/smithi077/crash 2023-01-24T17:33:17.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2023-01-24T17:33:17.434 INFO:tasks.ceph:Compressing logs... 2023-01-24T17:33:17.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-01-24T17:33:45.522 INFO:tasks.ceph:Archiving logs... 2023-01-24T17:33:45.524 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135503/remote/smithi077/log 2023-01-24T17:33:45.525 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/log/ceph -- . 2023-01-24T17:33:50.831 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-01-24T17:33:50.840 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2023-01-24T17:33:50.840 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2023-01-24T17:33:50.915 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-01-24T17:33:50.924 INFO:teuthology.task.clock:Checking final clock skew... 2023-01-24T17:33:50.924 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T17:33:50.950 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2023-01-24T17:33:50.955 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T17:33:50.955 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2023-01-24T17:33:50.955 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 38 +4062us[+4058us] +/- 100ms 2023-01-24T17:33:50.956 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -1384us[-1389us] +/- 51ms 2023-01-24T17:33:50.956 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 -807us[ -811us] +/- 73ms 2023-01-24T17:33:50.956 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T17:33:50.957 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-01-24T17:33:50.966 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-01-24T17:33:50.967 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-01-24T17:33:50.975 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-01-24T17:33:51.083 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T17:33:51.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T17:33:51.116 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-01-24T17:33:51.125 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-01-24T17:33:51.134 INFO:teuthology.task.internal:Duration was 954.261886 seconds 2023-01-24T17:33:51.135 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-01-24T17:33:51.143 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-01-24T17:33:51.144 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-01-24T17:33:51.203 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T17:33:51.383 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-01-24T17:33:51.384 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2023-01-24T17:33:51.384 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-01-24T17:33:51.406 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-01-24T17:33:51.407 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-01-24T17:33:51.488 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-01-24T17:33:51.489 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-01-24T17:33:51.597 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-01-24T17:33:51.608 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-01-24T17:33:51.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-01-24T17:33:51.636 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-01-24T17:33:51.646 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-01-24T17:33:51.703 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2023-01-24T17:33:51.717 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2023-01-24T17:33:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T17:33:51.774 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-01-24T17:33:51.783 INFO:teuthology.task.internal:Transferring archived files... 2023-01-24T17:33:51.784 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135503/remote/smithi077 2023-01-24T17:33:51.785 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2023-01-24T17:33:51.865 INFO:teuthology.task.internal:Removing archive directory... 2023-01-24T17:33:51.866 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2023-01-24T17:33:51.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-01-24T17:33:51.932 INFO:teuthology.task.internal:Not uploading archives. 2023-01-24T17:33:51.932 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-01-24T17:33:51.941 INFO:teuthology.task.internal:Tidying up after the test... 2023-01-24T17:33:51.942 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-01-24T17:33:51.979 INFO:teuthology.orchestra.run.smithi077.stdout: 525248 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 24 17:33 /home/ubuntu/cephtest 2023-01-24T17:33:51.980 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-01-24T17:33:52.002 INFO:teuthology.run:Summary data: description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} duration: 954.261885881424 flavor: default owner: scheduled_yuriw@teuthology success: true 2023-01-24T17:33:52.003 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T17:33:52.065 INFO:teuthology.run:pass