2022-11-28T01:04:36.644 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf 2022-11-28T01:04:36.646 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --name yuriw-2022-11-23_17:40:47-rgw-main-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2022-11-23_17:40:47-rgw-main-distro-default-smithi/7089245 --description rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests} -- /home/teuthworker/archive/yuriw-2022-11-23_17:40:47-rgw-main-distro-default-smithi/7089245/orig.config.yaml 2022-11-28T01:04:36.678 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-11-28T01:04:41.619 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-11-23_17:40:47-rgw-main-distro-default-smithi/7089245 branch: main description: rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests} email: ceph-qa@ceph.io first_in_suite: false job_id: '7089245' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-11-23_17:40:47-rgw-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: main ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw lc debug interval: 10 setgroup: ceph setuser: ceph storage classes: LUKEWARM, FROZEN mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 697b0d99d9a52df92b20c8e053dcd46c75f364a9 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 697b0d99d9a52df92b20c8e053dcd46c75f364a9 rgw: frontend: beast storage classes: LUKEWARM, FROZEN s3tests: force-branch: ceph-master selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 697b0d99d9a52df92b20c8e053dcd46c75f364a9 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - osd.2 - mgr.0 - client.0 sha1: 697b0d99d9a52df92b20c8e053dcd46c75f364a9 sleep_before_teardown: 0 suite: rgw suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_697b0d99d9a52df92b20c8e053dcd46c75f364a9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 697b0d99d9a52df92b20c8e053dcd46c75f364a9 targets: smithi003.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCl6Xr36MljkuCgtHHJ/YjyJonBGGRiSMd8PwWDqax4BJXiLfifmPZV3p0oZyBwZvX/UwomH4SFk4j7ozMYG1BgG2L5u+/wrjnR77bUaNADcQ7V2hXnv4a7zE9h49woOThJra+237YGTt2WH8fuixZHe+JeRGnDgn10dNxO14NSYJygTaVcx9exe+sHFSf7g2QZRpHgS0ANaN2OFP7F9LqFkvJJGfoa5lLrn/KSIY1FUITr5rpx3fEQ6F29H2513mfk4hr95CEqPm6k3gL34+wa4/WpdFFtZZamsMuz5GN+76cCgT8gOz6rxSS3g1/18qoSokrmp4vMJVyzg2dG7YGpo8BsXHvFQjW3KG3fnvvmVpL+upCg2o5uJE9iD0nOZFfEHj+Z1Tn3OfgJjT1dVZyO9roluNxlozcKLoXa/N9w1epuspCccis2IFir4EaiHILYC9+xmS/bn6N2ruPHFAkW/KjHzeGiav1+HflQzoo6d8qYHcE7v5kGw3waPyiTThc= tasks: - install: null - ceph: null - rgw: - client.0 - s3tests: client.0: extra_attrs: - lifecycle lc_debug_interval: 10 rgw_server: client.0 storage classes: LUKEWARM, FROZEN teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 4da97cf64e542f347ec47b7bdbe5eca99759f9b7 timestamp: 2022-11-23_17:40:47 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23126 2022-11-28T01:04:41.620 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_697b0d99d9a52df92b20c8e053dcd46c75f364a9/qa; will attempt to use it 2022-11-28T01:04:41.621 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_697b0d99d9a52df92b20c8e053dcd46c75f364a9/qa/tasks 2022-11-28T01:04:41.621 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-11-28T01:04:41.623 INFO:teuthology.task.internal:Checking packages... 2022-11-28T01:04:41.650 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '697b0d99d9a52df92b20c8e053dcd46c75f364a9' 2022-11-28T01:04:41.650 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-11-28T01:04:41.651 INFO:teuthology.packaging:ref: None 2022-11-28T01:04:41.651 INFO:teuthology.packaging:tag: None 2022-11-28T01:04:41.652 INFO:teuthology.packaging:branch: main 2022-11-28T01:04:41.652 INFO:teuthology.packaging:sha1: 697b0d99d9a52df92b20c8e053dcd46c75f364a9 2022-11-28T01:04:41.652 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=main 2022-11-28T01:04:56.518 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-995.gaf6ed506 2022-11-28T01:04:56.519 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-11-28T01:04:56.526 INFO:teuthology.task.internal:no buildpackages task found 2022-11-28T01:04:56.526 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-11-28T01:04:56.533 INFO:teuthology.task.internal:Saving configuration 2022-11-28T01:04:56.549 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-11-28T01:04:56.556 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-11-28T01:05:00.422 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-11-23_17:40:47-rgw-main-distro-default-smithi/7089245', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-11-28 00:57:47.942818', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWsIm84BGcxbRnGSZgwx9T+yJnh+JHwvlyRgfkYVxzlC5ezFhSFALjIzy7pWwECx01FB5r3D9OKDXaekgARggg='} 2022-11-28T01:05:00.423 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-11-28T01:05:00.433 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'osd.2', 'mgr.0', 'client.0'] 2022-11-28T01:05:00.434 INFO:teuthology.run_tasks:Running task console_log... 2022-11-28T01:05:00.503 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd261811f28>, signals=[15]) 2022-11-28T01:05:00.504 INFO:teuthology.run_tasks:Running task internal.connect... 2022-11-28T01:05:00.512 INFO:teuthology.task.internal:Opening connections... 2022-11-28T01:05:00.513 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2022-11-28T01:05:00.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-11-28T01:05:00.614 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-11-28T01:05:00.622 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2022-11-28T01:05:00.747 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2022-11-28T01:05:00.747 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2022-11-28T01:05:00.808 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2022-11-28T01:05:00.808 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="8" 2022-11-28T01:05:00.808 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2022-11-28T01:05:00.809 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2022-11-28T01:05:00.809 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="8" 2022-11-28T01:05:00.809 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el8" 2022-11-28T01:05:00.809 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 8" 2022-11-28T01:05:00.810 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2022-11-28T01:05:00.810 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-11-28T01:05:00.810 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2022-11-28T01:05:00.810 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-11-28T01:05:00.810 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-11-28T01:05:00.811 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-11-28T01:05:00.812 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2022-11-28T01:05:05.836 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-11-28T01:05:05.856 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-11-28T01:05:05.871 INFO:teuthology.task.internal:Checking for old test directory... 2022-11-28T01:05:05.872 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2022-11-28T01:05:05.893 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-11-28T01:05:05.905 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-11-28T01:05:05.906 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2022-11-28T01:05:05.975 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-11-28T01:05:11.018 INFO:teuthology.run_tasks:Running task kernel... 2022-11-28T01:05:11.039 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-11-28T01:05:11.040 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-11-28T01:05:11.041 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2022-11-28T01:05:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-28T01:05:11.063 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2022-11-28T01:05:11.122 INFO:teuthology.orchestra.run.smithi003.stdout:4.18.0-408.el8.x86_64 2022-11-28T01:05:11.123 INFO:teuthology.task.kernel:Running kernel on smithi003: 4.18.0-408.el8.x86_64 2022-11-28T01:05:11.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2022-11-28T01:05:14.210 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-8 - AppStream 39 MB/s | 26 MB 00:00 2022-11-28T01:05:19.718 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-8 - BaseOS 43 MB/s | 26 MB 00:00 2022-11-28T01:05:23.934 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-8 - Extras 162 kB/s | 18 kB 00:00 2022-11-28T01:05:24.302 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-8 - PowerTools 22 MB/s | 5.3 MB 00:00 2022-11-28T01:05:27.777 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2022-11-28T01:05:29.568 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 13 MB 00:01 2022-11-28T01:05:32.985 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 310 kB/s | 24 kB 00:00 2022-11-28T01:05:36.133 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2022-11-28T01:05:36.134 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-11-28T01:05:36.274 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:05:36.276 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2022-11-28T01:05:36.276 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:05:36.378 DEBUG:teuthology.orchestra.run.smithi003:> echo no | sudo yum reinstall kernel || true 2022-11-28T01:05:36.990 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:04 ago on Mon 28 Nov 2022 01:05:32 AM UTC. 2022-11-28T01:05:38.145 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:05:38.147 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:05:38.147 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2022-11-28T01:05:38.147 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:05:38.147 INFO:teuthology.orchestra.run.smithi003.stdout:Reinstalling: 2022-11-28T01:05:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-11-28T01:05:38.148 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:05:38.148 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:05:38.148 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:05:38.149 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:05:38.149 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 8.5 M 2022-11-28T01:05:38.149 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 0 2022-11-28T01:05:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:Operation aborted. 2022-11-28T01:05:38.219 INFO:teuthology.orchestra.run.smithi003.stdout:Is this ok [y/N]: 2022-11-28T01:05:38.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum reinstall -y kernel || true 2022-11-28T01:05:38.830 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:06 ago on Mon 28 Nov 2022 01:05:32 AM UTC. 2022-11-28T01:05:39.998 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:05:39.999 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:05:39.999 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2022-11-28T01:05:39.999 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:05:40.000 INFO:teuthology.orchestra.run.smithi003.stdout:Reinstalling: 2022-11-28T01:05:40.000 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-11-28T01:05:40.000 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:05:40.000 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:05:40.001 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:05:40.001 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:05:40.001 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 8.5 M 2022-11-28T01:05:40.001 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 0 2022-11-28T01:05:40.002 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:05:40.228 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-4.18.0-408.el8.x86_64.rpm 38 MB/s | 8.5 MB 00:00 2022-11-28T01:05:40.229 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:05:40.229 INFO:teuthology.orchestra.run.smithi003.stdout:Total 37 MB/s | 8.5 MB 00:00 2022-11-28T01:05:40.330 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:05:40.768 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:05:40.769 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:05:40.941 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:05:40.959 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:05:41.248 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:05:41.482 INFO:teuthology.orchestra.run.smithi003.stdout: Reinstalling : kernel-4.18.0-408.el8.x86_64 1/2 2022-11-28T01:05:41.482 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : kernel-4.18.0-408.el8.x86_64 2/2 2022-11-28T01:05:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-4.18.0-408.el8.x86_64 2/2 2022-11-28T01:05:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 1/2 2022-11-28T01:05:42.360 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 2/2 2022-11-28T01:05:42.360 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:05:42.361 INFO:teuthology.orchestra.run.smithi003.stdout:Reinstalled: 2022-11-28T01:05:42.361 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-4.18.0-408.el8.x86_64 2022-11-28T01:05:42.361 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:05:42.362 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:05:42.461 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel --last | head -n 1 2022-11-28T01:05:42.600 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-4.18.0-408.el8.x86_64 Mon 28 Nov 2022 01:05:41 AM UTC 2022-11-28T01:05:42.601 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-408.el8.x86_64 2022-11-28T01:05:42.602 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-11-28T01:05:42.602 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2022-11-28T01:05:42.602 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-11-28T01:05:42.639 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2022-11-28T01:05:42.652 DEBUG:teuthology.parallel:result is None 2022-11-28T01:05:42.653 INFO:teuthology.run_tasks:Running task internal.base... 2022-11-28T01:05:42.661 INFO:teuthology.task.internal:Creating test directory... 2022-11-28T01:05:42.662 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-11-28T01:05:42.710 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-11-28T01:05:42.718 INFO:teuthology.run_tasks:Running task internal.archive... 2022-11-28T01:05:42.745 INFO:teuthology.task.internal:Creating archive directory... 2022-11-28T01:05:42.745 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-11-28T01:05:42.779 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-11-28T01:05:42.787 INFO:teuthology.task.internal:Enabling coredump saving... 2022-11-28T01:05:42.788 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-11-28T01:05:42.844 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-11-28T01:05:42.854 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-11-28T01:05:42.856 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-11-28T01:05:42.865 INFO:teuthology.task.internal:Configuring sudo... 2022-11-28T01:05:42.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-11-28T01:05:42.925 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-11-28T01:05:42.935 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-11-28T01:05:42.936 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-11-28T01:05:42.983 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-11-28T01:05:44.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-11-28T01:05:44.265 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-11-28T01:05:44.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-11-28T01:05:44.401 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:05:44.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-11-28T01:05:44.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2022-11-28T01:05:44.603 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-11-28T01:05:44.878 INFO:teuthology.run_tasks:Running task internal.timer... 2022-11-28T01:05:44.890 INFO:teuthology.task.internal:Starting timer... 2022-11-28T01:05:44.891 INFO:teuthology.run_tasks:Running task pcp... 2022-11-28T01:05:44.911 INFO:teuthology.run_tasks:Running task selinux... 2022-11-28T01:05:44.928 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-11-28T01:05:44.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2022-11-28T01:05:45.016 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2022-11-28T01:05:45.018 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-11-28T01:05:45.019 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2022-11-28T01:05:45.047 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2022-11-28T01:05:45.048 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive'} 2022-11-28T01:05:45.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-11-28T01:05:45.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-28T01:05:45.117 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-11-28T01:05:45.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2022-11-28T01:05:45.198 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-11-28T01:05:45.216 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2022-11-28T01:05:45.225 INFO:teuthology.repo_utils:Fetching main from origin 2022-11-28T01:05:45.386 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2022-11-28T01:05:45.410 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-11-28T01:05:45.412 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2022-11-28T01:13:23.264 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com')] 2022-11-28T01:13:23.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2022-11-28T01:13:23.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-11-28T01:13:28.366 DEBUG:teuthology.orchestra.run.smithi003:> true 2022-11-28T01:13:28.443 INFO:teuthology.run_tasks:Running task clock... 2022-11-28T01:13:28.455 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-11-28T01:13:28.456 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-11-28T01:13:28.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-11-28T01:13:28.523 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-11-28T01:13:28.540 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-11-28T01:13:28.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2022-11-28T01:13:28.621 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2022-11-28T01:13:28.635 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-11-28T01:13:28.651 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-11-28T01:13:28.725 INFO:teuthology.orchestra.run.smithi003.stderr:bash: ntpq: command not found 2022-11-28T01:13:28.728 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-11-28T01:13:28.728 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2022-11-28T01:13:28.729 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-11-28T01:13:28.729 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-11-28T01:13:28.729 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-11-28T01:13:28.729 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-11-28T01:13:28.731 INFO:teuthology.run_tasks:Running task install... 2022-11-28T01:13:28.750 DEBUG:teuthology.task.install:project ceph 2022-11-28T01:13:28.751 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9'}} 2022-11-28T01:13:28.751 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9'} 2022-11-28T01:13:28.752 INFO:teuthology.task.install:Using flavor: default 2022-11-28T01:13:28.765 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-11-28T01:13:28.766 INFO:teuthology.task.install:extra packages: [] 2022-11-28T01:13:28.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo lsb_release -is 2022-11-28T01:13:28.810 INFO:teuthology.orchestra.run.smithi003.stdout:CentOSStream 2022-11-28T01:13:28.812 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-11-28T01:13:28.813 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9', 'tag': None, 'wait_for_package': False} 2022-11-28T01:13:28.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo lsb_release -is 2022-11-28T01:13:28.890 INFO:teuthology.orchestra.run.smithi003.stdout:CentOSStream 2022-11-28T01:13:28.891 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-11-28T01:13:28.892 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=697b0d99d9a52df92b20c8e053dcd46c75f364a9 2022-11-28T01:13:34.054 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/697b0d99d9a52df92b20c8e053dcd46c75f364a9/centos/8/flavors/default/ 2022-11-28T01:13:34.055 INFO:teuthology.task.install.rpm:Package version is 18.0.0-594.g697b0d99 2022-11-28T01:13:39.171 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/697b0d99d9a52df92b20c8e053dcd46c75f364a9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/697b0d99d9a52df92b20c8e053dcd46c75f364a9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/697b0d99d9a52df92b20c8e053dcd46c75f364a9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-11-28T01:13:39.172 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:13:39.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-11-28T01:13:39.206 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-11-28T01:13:39.207 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/697b0d99d9a52df92b20c8e053dcd46c75f364a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-11-28T01:13:39.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-11-28T01:13:39.358 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-11-28T01:13:39.427 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2022-11-28T01:13:39.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2022-11-28T01:13:39.697 INFO:teuthology.orchestra.run.smithi003.stdout:64 files removed 2022-11-28T01:13:39.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw 2022-11-28T01:13:40.388 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 291 kB/s | 81 kB 00:00 2022-11-28T01:13:40.656 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 41 kB/s | 9.4 kB 00:00 2022-11-28T01:13:40.948 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 5.7 kB/s | 1.5 kB 00:00 2022-11-28T01:13:41.634 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-8 - AppStream 39 MB/s | 26 MB 00:00 2022-11-28T01:13:47.217 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-8 - BaseOS 38 MB/s | 26 MB 00:00 2022-11-28T01:13:51.466 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-8 - Extras 134 kB/s | 18 kB 00:00 2022-11-28T01:13:51.918 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-8 - PowerTools 16 MB/s | 5.3 MB 00:00 2022-11-28T01:13:55.401 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2022-11-28T01:13:56.901 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 13 MB 00:01 2022-11-28T01:14:00.301 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 286 kB/s | 24 kB 00:00 2022-11-28T01:14:03.703 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:14:03.706 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:14:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2022-11-28T01:14:03.707 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:14:03.707 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:14:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:18.0.0-594.g697b0d99.el8 ceph 18 M 2022-11-28T01:14:03.707 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2022-11-28T01:14:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 3.6 M 2022-11-28T01:14:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 3.6 M 2022-11-28T01:14:03.708 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2022-11-28T01:14:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:18.0.0-594.g697b0d99.el8 ceph 5.2 M 2022-11-28T01:14:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:18.0.0-594.g697b0d99.el8 ceph 24 M 2022-11-28T01:14:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:18.0.0-594.g697b0d99.el8 ceph 24 k 2022-11-28T01:14:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-11-28T01:14:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 767 k 2022-11-28T01:14:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-11-28T01:14:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2022-11-28T01:14:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 474 k 2022-11-28T01:14:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-11-28T01:14:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 6.5 M 2022-11-28T01:14:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-11-28T01:14:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-11-28T01:14:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse x86_64 2:18.0.0-594.g697b0d99.el8 ceph 45 k 2022-11-28T01:14:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:18.0.0-594.g697b0d99.el8 ceph 109 k 2022-11-28T01:14:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:18.0.0-594.g697b0d99.el8 ceph 190 k 2022-11-28T01:14:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-11-28T01:14:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:18.0.0-594.g697b0d99.el8 ceph 371 k 2022-11-28T01:14:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:18.0.0-594.g697b0d99.el8 ceph 357 k 2022-11-28T01:14:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:18.0.0-594.g697b0d99.el8 ceph 107 k 2022-11-28T01:14:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-11-28T01:14:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:14:03.717 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:14:03.717 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:14:03.718 INFO:teuthology.orchestra.run.smithi003.stdout:Install 21 Packages 2022-11-28T01:14:03.718 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 2 Packages 2022-11-28T01:14:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:14:03.720 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 67 M 2022-11-28T01:14:03.720 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:14:04.411 INFO:teuthology.orchestra.run.smithi003.stdout:(1/23): ceph-base-18.0.0-594.g697b0d99.el8.x86_ 7.4 MB/s | 5.2 MB 00:00 2022-11-28T01:14:04.460 INFO:teuthology.orchestra.run.smithi003.stdout:(2/23): ceph-selinux-18.0.0-594.g697b0d99.el8.x 489 kB/s | 24 kB 00:00 2022-11-28T01:14:04.552 INFO:teuthology.orchestra.run.smithi003.stdout:(3/23): libcephfs2-18.0.0-594.g697b0d99.el8.x86 8.2 MB/s | 767 kB 00:00 2022-11-28T01:14:04.636 INFO:teuthology.orchestra.run.smithi003.stdout:(4/23): libradosstriper1-18.0.0-594.g697b0d99.e 5.6 MB/s | 474 kB 00:00 2022-11-28T01:14:05.245 INFO:teuthology.orchestra.run.smithi003.stdout:(5/23): librgw2-18.0.0-594.g697b0d99.el8.x86_64 11 MB/s | 6.5 MB 00:00 2022-11-28T01:14:05.317 INFO:teuthology.orchestra.run.smithi003.stdout:(6/23): python3-ceph-argparse-18.0.0-594.g697b0 672 kB/s | 45 kB 00:00 2022-11-28T01:14:05.387 INFO:teuthology.orchestra.run.smithi003.stdout:(7/23): python3-ceph-common-18.0.0-594.g697b0d9 1.4 MB/s | 109 kB 00:00 2022-11-28T01:14:05.462 INFO:teuthology.orchestra.run.smithi003.stdout:(8/23): python3-cephfs-18.0.0-594.g697b0d99.el8 2.5 MB/s | 190 kB 00:00 2022-11-28T01:14:05.554 INFO:teuthology.orchestra.run.smithi003.stdout:(9/23): python3-rados-18.0.0-594.g697b0d99.el8. 4.0 MB/s | 371 kB 00:00 2022-11-28T01:14:05.638 INFO:teuthology.orchestra.run.smithi003.stdout:(10/23): python3-rbd-18.0.0-594.g697b0d99.el8.x 4.2 MB/s | 357 kB 00:00 2022-11-28T01:14:05.713 INFO:teuthology.orchestra.run.smithi003.stdout:(11/23): python3-rgw-18.0.0-594.g697b0d99.el8.x 1.4 MB/s | 107 kB 00:00 2022-11-28T01:14:05.948 INFO:teuthology.orchestra.run.smithi003.stdout:(12/23): ceph-radosgw-18.0.0-594.g697b0d99.el8. 8.2 MB/s | 18 MB 00:02 2022-11-28T01:14:05.980 INFO:teuthology.orchestra.run.smithi003.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.3 MB/s | 354 kB 00:00 2022-11-28T01:14:06.006 INFO:teuthology.orchestra.run.smithi003.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.8 MB/s | 44 kB 00:00 2022-11-28T01:14:06.031 INFO:teuthology.orchestra.run.smithi003.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2022-11-28T01:14:06.056 INFO:teuthology.orchestra.run.smithi003.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.3 MB/s | 259 kB 00:00 2022-11-28T01:14:06.282 INFO:teuthology.orchestra.run.smithi003.stdout:(17/23): ceph-common-18.0.0-594.g697b0d99.el8.x 9.5 MB/s | 24 MB 00:02 2022-11-28T01:14:06.365 INFO:teuthology.orchestra.run.smithi003.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 917 kB/s | 306 kB 00:00 2022-11-28T01:14:06.398 INFO:teuthology.orchestra.run.smithi003.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 172 kB/s | 59 kB 00:00 2022-11-28T01:14:06.416 INFO:teuthology.orchestra.run.smithi003.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 561 kB/s | 75 kB 00:00 2022-11-28T01:14:06.566 INFO:teuthology.orchestra.run.smithi003.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 8.7 MB/s | 1.7 MB 00:00 2022-11-28T01:14:06.727 INFO:teuthology.orchestra.run.smithi003.stdout:(22/23): librados2-18.0.0-594.g697b0d99.el8.x86 11 MB/s | 3.6 MB 00:00 2022-11-28T01:14:06.791 INFO:teuthology.orchestra.run.smithi003.stdout:(23/23): librbd1-18.0.0-594.g697b0d99.el8.x86_6 9.5 MB/s | 3.6 MB 00:00 2022-11-28T01:14:06.847 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:14:06.848 INFO:teuthology.orchestra.run.smithi003.stdout:Total 22 MB/s | 67 MB 00:03 2022-11-28T01:14:06.848 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:14:06.886 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:14:06.886 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:14:07.275 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:14:07.278 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:14:08.526 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:14:08.688 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2022-11-28T01:14:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2022-11-28T01:14:09.090 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-11-28T01:14:09.531 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-11-28T01:14:09.748 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:18.0.0-594.g697b0d99.el8.x86_64 4/25 2022-11-28T01:14:09.846 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:18.0.0-594.g697b0d99.el8.x86_64 4/25 2022-11-28T01:14:10.333 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:18.0.0-594.g697b0d99.el8.x86_64 5/25 2022-11-28T01:14:10.540 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 6/25 2022-11-28T01:14:10.948 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 6/25 2022-11-28T01:14:11.098 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 7/25 2022-11-28T01:14:11.148 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 7/25 2022-11-28T01:14:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2022-11-28T01:14:11.506 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2022-11-28T01:14:11.568 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2022-11-28T01:14:11.743 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:18.0.0-594.g697b0d99.el8.x 10/25 2022-11-28T01:14:11.943 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:18.0.0-594.g697b0d99.el8.x86_64 11/25 2022-11-28T01:14:12.194 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:18.0.0-594.g697b0d99.el8.x86_64 12/25 2022-11-28T01:14:12.348 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:18.0.0-594.g697b0d99.el8.x86_64 13/25 2022-11-28T01:14:12.423 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:18.0.0-594.g697b0d99.el8.x86_64 13/25 2022-11-28T01:14:12.642 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2022-11-28T01:14:13.425 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2022-11-28T01:14:13.573 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:18.0.0-594.g697b0d99.el8.x86_64 16/25 2022-11-28T01:14:13.645 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:18.0.0-594.g697b0d99.el8.x86_64 16/25 2022-11-28T01:14:13.848 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:18.0.0-594.g697b0d99.el8.x86_64 17/25 2022-11-28T01:14:14.025 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-11-28T01:14:14.206 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:18.0.0-594.g697b0d99.el8.x86 19/25 2022-11-28T01:14:18.120 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 20/25 2022-11-28T01:14:18.340 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 20/25 2022-11-28T01:14:18.957 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 20/25 2022-11-28T01:14:19.185 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 21/25 2022-11-28T01:14:19.263 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 21/25 2022-11-28T01:14:19.389 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:18.0.0-594.g697b0d99.el8.x86_64 22/25 2022-11-28T01:14:39.658 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:18.0.0-594.g697b0d99.el8.x86_64 22/25 2022-11-28T01:14:39.659 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2022-11-28T01:14:39.659 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2022-11-28T01:14:39.660 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /dev 2022-11-28T01:14:39.660 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /run 2022-11-28T01:14:39.660 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2022-11-28T01:14:39.660 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2022-11-28T01:14:39.661 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2022-11-28T01:14:39.661 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2022-11-28T01:14:39.661 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /dev 2022-11-28T01:14:39.661 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:14:41.573 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:18.0.0-594.g697b0d99.el8.x86_64 23/25 2022-11-28T01:14:41.595 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-594.g697b0d99.el8.x86_64 23/25 2022-11-28T01:14:41.595 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-11-28T01:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-11-28T01:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-11-28T01:14:42.510 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-11-28T01:14:42.511 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 1/25 2022-11-28T01:14:42.511 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 2/25 2022-11-28T01:14:42.512 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:18.0.0-594.g697b0d99.el8.x86_64 3/25 2022-11-28T01:14:42.512 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:18.0.0-594.g697b0d99.el8.x86_64 4/25 2022-11-28T01:14:42.513 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 5/25 2022-11-28T01:14:42.522 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:18.0.0-594.g697b0d99.el8.x86_64 6/25 2022-11-28T01:14:42.523 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:18.0.0-594.g697b0d99.el8.x86_64 7/25 2022-11-28T01:14:42.523 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:18.0.0-594.g697b0d99.el8.x 8/25 2022-11-28T01:14:42.523 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:18.0.0-594.g697b0d99.el8.x86 9/25 2022-11-28T01:14:42.524 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:18.0.0-594.g697b0d99.el8.x86_64 10/25 2022-11-28T01:14:42.524 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:18.0.0-594.g697b0d99.el8.x86_64 11/25 2022-11-28T01:14:42.525 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:18.0.0-594.g697b0d99.el8.x86_64 12/25 2022-11-28T01:14:42.526 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:18.0.0-594.g697b0d99.el8.x86_64 13/25 2022-11-28T01:14:42.526 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-11-28T01:14:42.527 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-11-28T01:14:42.527 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-11-28T01:14:42.527 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-11-28T01:14:42.528 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-11-28T01:14:42.528 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2022-11-28T01:14:42.528 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2022-11-28T01:14:42.528 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2022-11-28T01:14:42.529 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:18.0.0-594.g697b0d99.el8.x86_64 22/25 2022-11-28T01:14:42.529 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-11-28T01:14:42.529 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 24/25 2022-11-28T01:14:43.487 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-11-28T01:14:43.487 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:14:43.488 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2022-11-28T01:14:43.488 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.488 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.488 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:14:43.489 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.489 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.489 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.489 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.490 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-11-28T01:14:43.490 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.490 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.2-3.el8.x86_64 2022-11-28T01:14:43.490 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-11-28T01:14:43.490 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.491 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-11-28T01:14:43.491 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.491 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.3.1-3.el8.x86_64 2022-11-28T01:14:43.491 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-11-28T01:14:43.492 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.492 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.492 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.492 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-11-28T01:14:43.493 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.493 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.493 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:43.493 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.13.0-2.el8.x86_64 2022-11-28T01:14:43.494 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:14:43.494 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:14:43.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test 2022-11-28T01:14:44.507 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:44 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:14:45.673 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:14:45.674 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:14:45.675 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2022-11-28T01:14:45.675 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:14:45.676 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:14:45.676 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:18.0.0-594.g697b0d99.el8 ceph 55 M 2022-11-28T01:14:45.677 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2022-11-28T01:14:45.678 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2022-11-28T01:14:45.678 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:18.0.0-594.g697b0d99.el8 ceph 137 k 2022-11-28T01:14:45.678 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-11-28T01:14:45.679 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-11-28T01:14:45.679 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2022-11-28T01:14:45.680 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:14:45.680 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:14:45.681 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:14:45.681 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2022-11-28T01:14:45.682 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:14:45.683 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 56 M 2022-11-28T01:14:45.684 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 241 M 2022-11-28T01:14:45.684 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:14:45.745 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): jq-1.6-6.el8.x86_64.rpm 2.9 MB/s | 202 kB 00:00 2022-11-28T01:14:45.771 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.6 MB/s | 187 kB 00:00 2022-11-28T01:14:45.822 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 6.1 MB/s | 323 kB 00:00 2022-11-28T01:14:45.854 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.2 MB/s | 71 kB 00:00 2022-11-28T01:14:45.946 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): libcephsqlite-18.0.0-594.g697b0d99.el8.x 509 kB/s | 137 kB 00:00 2022-11-28T01:14:48.871 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): ceph-test-18.0.0-594.g697b0d99.el8.x86_6 17 MB/s | 55 MB 00:03 2022-11-28T01:14:48.873 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:14:48.874 INFO:teuthology.orchestra.run.smithi003.stdout:Total 17 MB/s | 56 MB 00:03 2022-11-28T01:14:48.917 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:14:48.940 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:14:48.941 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:14:49.261 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:14:49.264 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:14:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:14:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2022-11-28T01:14:50.143 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-11-28T01:14:50.334 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-11-28T01:14:50.401 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-11-28T01:14:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-6.el8.x86_64 4/6 2022-11-28T01:14:50.742 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:18.0.0-594.g697b0d99.el8.x86_64 5/6 2022-11-28T01:14:56.258 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:18.0.0-594.g697b0d99.el8.x86_64 5/6 2022-11-28T01:14:56.489 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:18.0.0-594.g697b0d99.el8.x86_64 6/6 2022-11-28T01:14:56.732 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-2:18.0.0-594.g697b0d99.el8.x86_64 6/6 2022-11-28T01:14:56.732 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:18.0.0-594.g697b0d99.el8.x86_64 1/6 2022-11-28T01:14:56.732 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:18.0.0-594.g697b0d99.el8.x86_64 2/6 2022-11-28T01:14:56.733 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-6.el8.x86_64 3/6 2022-11-28T01:14:56.733 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-11-28T01:14:56.733 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-11-28T01:14:57.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2022-11-28T01:14:57.218 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:14:57.218 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:14:57.219 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:57.219 INFO:teuthology.orchestra.run.smithi003.stdout: jq-1.6-6.el8.x86_64 2022-11-28T01:14:57.219 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:14:57.220 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-11-28T01:14:57.220 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-1.el8.x86_64 2022-11-28T01:14:57.221 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-11-28T01:14:57.221 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:14:57.221 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:14:57.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph 2022-11-28T01:14:58.041 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:58 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:14:59.213 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:14:59.217 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2022-11-28T01:14:59.217 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2022-11-28T01:14:59.218 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2022-11-28T01:14:59.218 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:14:59.218 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:18.0.0-594.g697b0d99.el8 ceph 6.4 k 2022-11-28T01:14:59.219 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2022-11-28T01:14:59.219 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:18.0.0-594.g697b0d99.el8 ceph 2.3 M 2022-11-28T01:14:59.219 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:18.0.0-594.g697b0d99.el8 ceph 1.5 M 2022-11-28T01:14:59.219 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 234 k 2022-11-28T01:14:59.220 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:18.0.0-594.g697b0d99.el8 ceph 3.9 M 2022-11-28T01:14:59.220 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:18.0.0-594.g697b0d99.el8 ceph 17 M 2022-11-28T01:14:59.221 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-11-28T01:14:59.221 INFO:teuthology.orchestra.run.smithi003.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-11-28T01:14:59.221 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2022-11-28T01:14:59.221 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-11-28T01:14:59.222 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-11-28T01:14:59.222 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2022-11-28T01:14:59.222 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-11-28T01:14:59.222 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-11-28T01:14:59.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-11-28T01:14:59.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-11-28T01:14:59.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-11-28T01:14:59.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-11-28T01:14:59.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-11-28T01:14:59.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-11-28T01:14:59.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-11-28T01:14:59.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2022-11-28T01:14:59.225 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-11-28T01:14:59.225 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-11-28T01:14:59.225 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-11-28T01:14:59.225 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-11-28T01:14:59.226 INFO:teuthology.orchestra.run.smithi003.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-11-28T01:14:59.226 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-11-28T01:14:59.226 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-11-28T01:14:59.226 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-11-28T01:14:59.227 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-11-28T01:14:59.227 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-11-28T01:14:59.227 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:14:59.227 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:14:59.228 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2022-11-28T01:14:59.228 INFO:teuthology.orchestra.run.smithi003.stdout:Install 32 Packages 2022-11-28T01:14:59.228 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:14:59.229 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 28 M 2022-11-28T01:14:59.229 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 110 M 2022-11-28T01:14:59.229 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:14:59.356 INFO:teuthology.orchestra.run.smithi003.stdout:(1/32): ceph-18.0.0-594.g697b0d99.el8.x86_64.rp 48 kB/s | 6.4 kB 00:00 2022-11-28T01:14:59.589 INFO:teuthology.orchestra.run.smithi003.stdout:(2/32): ceph-mds-18.0.0-594.g697b0d99.el8.x86_6 6.2 MB/s | 2.3 MB 00:00 2022-11-28T01:14:59.648 INFO:teuthology.orchestra.run.smithi003.stdout:(3/32): ceph-mgr-18.0.0-594.g697b0d99.el8.x86_6 3.6 MB/s | 1.5 MB 00:00 2022-11-28T01:14:59.740 INFO:teuthology.orchestra.run.smithi003.stdout:(4/32): ceph-mgr-modules-core-18.0.0-594.g697b0 2.5 MB/s | 234 kB 00:00 2022-11-28T01:14:59.798 INFO:teuthology.orchestra.run.smithi003.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2022-11-28T01:14:59.824 INFO:teuthology.orchestra.run.smithi003.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2022-11-28T01:14:59.849 INFO:teuthology.orchestra.run.smithi003.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.1 MB/s | 103 kB 00:00 2022-11-28T01:14:59.874 INFO:teuthology.orchestra.run.smithi003.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2022-11-28T01:14:59.908 INFO:teuthology.orchestra.run.smithi003.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2022-11-28T01:14:59.925 INFO:teuthology.orchestra.run.smithi003.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 14 MB/s | 237 kB 00:00 2022-11-28T01:14:59.958 INFO:teuthology.orchestra.run.smithi003.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 16 MB/s | 559 kB 00:00 2022-11-28T01:14:59.984 INFO:teuthology.orchestra.run.smithi003.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2022-11-28T01:15:00.117 INFO:teuthology.orchestra.run.smithi003.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 328 kB/s | 44 kB 00:00 2022-11-28T01:15:00.226 INFO:teuthology.orchestra.run.smithi003.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.7 MB/s | 185 kB 00:00 2022-11-28T01:15:00.301 INFO:teuthology.orchestra.run.smithi003.stdout:(15/32): ceph-mon-18.0.0-594.g697b0d99.el8.x86_ 4.1 MB/s | 3.9 MB 00:00 2022-11-28T01:15:00.326 INFO:teuthology.orchestra.run.smithi003.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.7 MB/s | 173 kB 00:00 2022-11-28T01:15:00.527 INFO:teuthology.orchestra.run.smithi003.stdout:(17/32): ceph-osd-18.0.0-594.g697b0d99.el8.x86_ 18 MB/s | 17 MB 00:00 2022-11-28T01:15:00.552 INFO:teuthology.orchestra.run.smithi003.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 49 kB/s | 11 kB 00:00 2022-11-28T01:15:00.578 INFO:teuthology.orchestra.run.smithi003.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 351 kB/s | 18 kB 00:00 2022-11-28T01:15:00.619 INFO:teuthology.orchestra.run.smithi003.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.2 MB/s | 384 kB 00:00 2022-11-28T01:15:00.644 INFO:teuthology.orchestra.run.smithi003.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 879 kB/s | 59 kB 00:00 2022-11-28T01:15:00.670 INFO:teuthology.orchestra.run.smithi003.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2022-11-28T01:15:00.695 INFO:teuthology.orchestra.run.smithi003.stdout:(23/32): python3-logutils-0.3.5-11.el8.noarch.r 341 kB/s | 49 kB 00:00 2022-11-28T01:15:00.720 INFO:teuthology.orchestra.run.smithi003.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 206 kB/s | 16 kB 00:00 2022-11-28T01:15:00.738 INFO:teuthology.orchestra.run.smithi003.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 277 kB/s | 19 kB 00:00 2022-11-28T01:15:00.754 INFO:teuthology.orchestra.run.smithi003.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 404 kB/s | 24 kB 00:00 2022-11-28T01:15:00.771 INFO:teuthology.orchestra.run.smithi003.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 573 kB/s | 29 kB 00:00 2022-11-28T01:15:00.788 INFO:teuthology.orchestra.run.smithi003.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 546 kB/s | 27 kB 00:00 2022-11-28T01:15:00.859 INFO:teuthology.orchestra.run.smithi003.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.2 MB/s | 85 kB 00:00 2022-11-28T01:15:00.889 INFO:teuthology.orchestra.run.smithi003.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2022-11-28T01:15:00.906 INFO:teuthology.orchestra.run.smithi003.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 498 kB/s | 23 kB 00:00 2022-11-28T01:15:00.931 INFO:teuthology.orchestra.run.smithi003.stdout:(32/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.3 MB/s | 241 kB 00:00 2022-11-28T01:15:00.934 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:15:00.935 INFO:teuthology.orchestra.run.smithi003.stdout:Total 16 MB/s | 28 MB 00:01 2022-11-28T01:15:01.014 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:15:01.053 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:15:01.054 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:15:01.336 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:15:01.340 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:15:02.545 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:15:02.766 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-11-28T01:15:02.990 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-11-28T01:15:03.208 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-11-28T01:15:03.413 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-11-28T01:15:03.578 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-11-28T01:15:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-11-28T01:15:03.910 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-11-28T01:15:04.046 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-11-28T01:15:04.257 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-11-28T01:15:04.485 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-11-28T01:15:04.715 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-11-28T01:15:04.915 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-11-28T01:15:05.332 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-11-28T01:15:05.577 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-11-28T01:15:05.740 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-11-28T01:15:05.985 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-11-28T01:15:06.209 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-11-28T01:15:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-11-28T01:15:06.752 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-11-28T01:15:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-11-28T01:15:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-11-28T01:15:07.540 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-11-28T01:15:07.735 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-11-28T01:15:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-11-28T01:15:08.290 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-11-28T01:15:08.626 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-11-28T01:15:08.997 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-594.g697b0d99.el8.n 27/32 2022-11-28T01:15:09.178 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 28/32 2022-11-28T01:15:10.762 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 28/32 2022-11-28T01:15:10.944 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:18.0.0-594.g697b0d99.el8.x86_64 29/32 2022-11-28T01:15:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:18.0.0-594.g697b0d99.el8.x86_64 29/32 2022-11-28T01:15:11.531 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:18.0.0-594.g697b0d99.el8.x86_64 30/32 2022-11-28T01:15:11.756 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:18.0.0-594.g697b0d99.el8.x86_64 30/32 2022-11-28T01:15:11.947 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:18.0.0-594.g697b0d99.el8.x86_64 31/32 2022-11-28T01:15:11.966 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:18.0.0-594.g697b0d99.el8.x86_64 31/32 2022-11-28T01:15:12.079 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:18.0.0-594.g697b0d99.el8.x86_64 32/32 2022-11-28T01:15:12.471 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-2:18.0.0-594.g697b0d99.el8.x86_64 32/32 2022-11-28T01:15:12.471 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:18.0.0-594.g697b0d99.el8.x86_64 1/32 2022-11-28T01:15:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:18.0.0-594.g697b0d99.el8.x86_64 2/32 2022-11-28T01:15:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 3/32 2022-11-28T01:15:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:18.0.0-594.g697b0d99.el8.x86_64 4/32 2022-11-28T01:15:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:18.0.0-594.g697b0d99.el8.x86_64 5/32 2022-11-28T01:15:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-594.g697b0d99.el8.n 6/32 2022-11-28T01:15:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-11-28T01:15:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-11-28T01:15:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-11-28T01:15:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-11-28T01:15:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-11-28T01:15:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-11-28T01:15:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-11-28T01:15:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-11-28T01:15:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-11-28T01:15:12.476 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-11-28T01:15:12.476 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-11-28T01:15:12.477 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-11-28T01:15:12.477 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-11-28T01:15:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-11-28T01:15:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-11-28T01:15:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-11-28T01:15:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-11-28T01:15:12.479 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-11-28T01:15:12.479 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-11-28T01:15:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-11-28T01:15:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-11-28T01:15:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-11-28T01:15:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-11-28T01:15:12.482 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-11-28T01:15:12.482 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-11-28T01:15:13.377 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-11-28T01:15:13.377 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:13.377 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:15:13.377 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:15:13.378 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:15:13.378 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:15:13.378 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-28T01:15:13.378 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:15:13.379 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:15:13.379 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-11-28T01:15:13.379 INFO:teuthology.orchestra.run.smithi003.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-11-28T01:15:13.379 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-11-28T01:15:13.380 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-11-28T01:15:13.380 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-11-28T01:15:13.380 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-11-28T01:15:13.380 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-6.2-6.el8.noarch 2022-11-28T01:15:13.380 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-11-28T01:15:13.381 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-11-28T01:15:13.381 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.0.6-13.el8.noarch 2022-11-28T01:15:13.381 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-11-28T01:15:13.382 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-11-28T01:15:13.382 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-11-28T01:15:13.382 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-2.6-1.el8.noarch 2022-11-28T01:15:13.382 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-11-28T01:15:13.383 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.14-14.el8.noarch 2022-11-28T01:15:13.383 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2017.2-9.el8.noarch 2022-11-28T01:15:13.383 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-11-28T01:15:13.383 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-11-28T01:15:13.384 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-11-28T01:15:13.384 INFO:teuthology.orchestra.run.smithi003.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-11-28T01:15:13.384 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-11-28T01:15:13.384 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-11-28T01:15:13.385 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-11-28T01:15:13.385 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-11-28T01:15:13.385 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-11-28T01:15:13.386 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:13.386 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:15:13.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base 2022-11-28T01:15:14.252 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:14 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:15:15.301 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 is already installed. 2022-11-28T01:15:15.403 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:15:15.404 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2022-11-28T01:15:15.404 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:15:15.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install cephadm 2022-11-28T01:15:16.328 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:16 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:15:17.487 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:15:17.489 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:15:17.489 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2022-11-28T01:15:17.489 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:15:17.489 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:15:17.490 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 81 k 2022-11-28T01:15:17.490 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:17.490 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:15:17.490 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:15:17.490 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2022-11-28T01:15:17.491 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:17.491 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 81 k 2022-11-28T01:15:17.491 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 368 k 2022-11-28T01:15:17.491 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:15:18.001 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm-18.0.0-594.g697b0d99.el8.noarch.rpm 159 kB/s | 81 kB 00:00 2022-11-28T01:15:18.002 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:15:18.003 INFO:teuthology.orchestra.run.smithi003.stdout:Total 158 kB/s | 81 kB 00:00 2022-11-28T01:15:18.003 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:15:18.011 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:15:18.011 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:15:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:15:18.016 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:15:18.094 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:15:19.800 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:18.0.0-594.g697b0d99.el8.noarch 1/1 2022-11-28T01:15:19.939 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:18.0.0-594.g697b0d99.el8.noarch 1/1 2022-11-28T01:15:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:18.0.0-594.g697b0d99.el8.noarch 1/1 2022-11-28T01:15:20.536 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:18.0.0-594.g697b0d99.el8.noarch 1/1 2022-11-28T01:15:20.537 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:20.537 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:15:20.537 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-28T01:15:20.539 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:20.539 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:15:20.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache 2022-11-28T01:15:21.417 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:21 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:15:22.581 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:15:22.582 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:15:22.582 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2022-11-28T01:15:22.582 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:15:22.583 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:15:22.583 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-594.g697b0d99.el8 ceph 165 k 2022-11-28T01:15:22.583 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:22.583 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:15:22.584 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:15:22.584 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2022-11-28T01:15:22.584 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:22.584 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 165 k 2022-11-28T01:15:22.584 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 503 k 2022-11-28T01:15:22.585 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:15:23.108 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-18.0.0-594.g697b0d9 315 kB/s | 165 kB 00:00 2022-11-28T01:15:23.109 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:15:23.110 INFO:teuthology.orchestra.run.smithi003.stdout:Total 314 kB/s | 165 kB 00:00 2022-11-28T01:15:23.110 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:15:23.122 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:15:23.122 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:15:23.175 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:15:23.178 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:15:23.307 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:15:23.495 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-594.g697b0d99.e 1/1 2022-11-28T01:15:23.890 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-594.g697b0d99.e 1/1 2022-11-28T01:15:24.362 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-594.g697b0d99.e 1/1 2022-11-28T01:15:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:24.363 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:15:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:15:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:24.364 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:15:24.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr 2022-11-28T01:15:25.196 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:25 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:15:26.261 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 is already installed. 2022-11-28T01:15:26.364 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:15:26.365 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2022-11-28T01:15:26.365 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:15:26.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-dashboard 2022-11-28T01:15:27.601 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:27 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:15:28.761 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:15:28.762 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:15:28.763 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2022-11-28T01:15:28.763 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:15:28.763 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:15:28.764 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 1.7 M 2022-11-28T01:15:28.764 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2022-11-28T01:15:28.764 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 23 k 2022-11-28T01:15:28.765 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 14 k 2022-11-28T01:15:28.765 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2022-11-28T01:15:28.765 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-11-28T01:15:28.765 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-11-28T01:15:28.765 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:28.766 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:15:28.766 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:15:28.766 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2022-11-28T01:15:28.766 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:28.767 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.0 M 2022-11-28T01:15:28.768 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 7.8 M 2022-11-28T01:15:28.768 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:15:29.037 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): ceph-grafana-dashboards-18.0.0-594.g697b 84 kB/s | 23 kB 00:00 2022-11-28T01:15:29.062 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): ceph-prometheus-alerts-18.0.0-594.g697b0 49 kB/s | 14 kB 00:00 2022-11-28T01:15:29.113 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 567 kB/s | 43 kB 00:00 2022-11-28T01:15:29.229 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 201 kB/s | 34 kB 00:00 2022-11-28T01:15:29.405 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): ceph-mgr-dashboard-18.0.0-594.g697b0d99. 2.6 MB/s | 1.7 MB 00:00 2022-11-28T01:15:29.430 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 619 kB/s | 196 kB 00:00 2022-11-28T01:15:29.432 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:15:29.432 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.9 MB/s | 2.0 MB 00:00 2022-11-28T01:15:29.444 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:15:29.452 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:15:29.453 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:15:29.525 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:15:29.528 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:15:30.002 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-11-28T01:15:30.205 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-11-28T01:15:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-11-28T01:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-594.g697b0d99.el8.no 4/6 2022-11-28T01:15:30.919 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-594.g697b0d99.el8.n 5/6 2022-11-28T01:15:31.240 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-594.g697b0d99.el8.noarch 6/6 2022-11-28T01:15:31.485 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-594.g697b0d99.el8.noarch 6/6 2022-11-28T01:15:31.485 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-594.g697b0d99.el8.n 1/6 2022-11-28T01:15:31.485 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-594.g697b0d99.el8.noarch 2/6 2022-11-28T01:15:31.485 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-594.g697b0d99.el8.no 3/6 2022-11-28T01:15:31.486 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-11-28T01:15:31.486 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-11-28T01:15:31.919 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-11-28T01:15:31.920 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:31.920 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:15:31.920 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-28T01:15:31.920 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-28T01:15:31.921 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-28T01:15:31.921 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-11-28T01:15:31.921 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-11-28T01:15:31.921 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.4.1-12.el8.noarch 2022-11-28T01:15:31.922 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:31.922 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:15:32.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-diskprediction-local 2022-11-28T01:15:32.648 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:32 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:15:33.817 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:15:33.818 INFO:teuthology.orchestra.run.smithi003.stdout:===================================================================================================== 2022-11-28T01:15:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2022-11-28T01:15:33.819 INFO:teuthology.orchestra.run.smithi003.stdout:===================================================================================================== 2022-11-28T01:15:33.820 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:15:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 7.4 M 2022-11-28T01:15:33.821 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2022-11-28T01:15:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 8.5.0-17.el8 CentOS-BaseOS 645 k 2022-11-28T01:15:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 8.5.0-17.el8 CentOS-BaseOS 172 k 2022-11-28T01:15:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2022-11-28T01:15:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2022-11-28T01:15:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-11-28T01:15:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-11-28T01:15:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-11-28T01:15:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:33.825 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:15:33.825 INFO:teuthology.orchestra.run.smithi003.stdout:===================================================================================================== 2022-11-28T01:15:33.825 INFO:teuthology.orchestra.run.smithi003.stdout:Install 8 Packages 2022-11-28T01:15:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:33.826 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 35 M 2022-11-28T01:15:33.826 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 210 M 2022-11-28T01:15:33.827 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:15:33.977 INFO:teuthology.orchestra.run.smithi003.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2022-11-28T01:15:34.077 INFO:teuthology.orchestra.run.smithi003.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 37 MB/s | 3.7 MB 00:00 2022-11-28T01:15:34.111 INFO:teuthology.orchestra.run.smithi003.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2022-11-28T01:15:34.329 INFO:teuthology.orchestra.run.smithi003.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.6 MB/s | 4.9 MB 00:00 2022-11-28T01:15:34.362 INFO:teuthology.orchestra.run.smithi003.stdout:(5/8): libgfortran-8.5.0-17.el8.x86_64.rpm 19 MB/s | 645 kB 00:00 2022-11-28T01:15:34.387 INFO:teuthology.orchestra.run.smithi003.stdout:(6/8): libquadmath-8.5.0-17.el8.x86_64.rpm 6.7 MB/s | 172 kB 00:00 2022-11-28T01:15:34.529 INFO:teuthology.orchestra.run.smithi003.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 33 MB/s | 14 MB 00:00 2022-11-28T01:15:34.896 INFO:teuthology.orchestra.run.smithi003.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-594 6.9 MB/s | 7.4 MB 00:01 2022-11-28T01:15:34.898 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:15:34.899 INFO:teuthology.orchestra.run.smithi003.stdout:Total 33 MB/s | 35 MB 00:01 2022-11-28T01:15:35.135 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:15:35.154 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:15:35.154 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:15:35.505 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:15:35.508 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:15:36.068 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:15:36.210 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-8.5.0-17.el8.x86_64 1/8 2022-11-28T01:15:36.355 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libquadmath-8.5.0-17.el8.x86_64 1/8 2022-11-28T01:15:36.476 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-8.5.0-17.el8.x86_64 2/8 2022-11-28T01:15:37.104 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libgfortran-8.5.0-17.el8.x86_64 2/8 2022-11-28T01:15:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2022-11-28T01:15:37.934 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2022-11-28T01:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-11-28T01:15:38.942 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-11-28T01:15:39.293 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-11-28T01:15:39.517 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-11-28T01:15:41.395 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-11-28T01:15:42.709 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-11-28T01:15:42.875 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-594.g697b0d99 8/8 2022-11-28T01:15:43.164 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-594.g697b0d99 8/8 2022-11-28T01:15:43.165 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-594.g697b0d99 1/8 2022-11-28T01:15:43.165 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2022-11-28T01:15:43.166 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2022-11-28T01:15:43.167 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-11-28T01:15:43.167 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-11-28T01:15:43.167 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-11-28T01:15:43.168 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-8.5.0-17.el8.x86_64 7/8 2022-11-28T01:15:44.246 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-8.5.0-17.el8.x86_64 8/8 2022-11-28T01:15:44.246 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:44.247 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:15:44.247 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-28T01:15:44.247 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-8.5.0-17.el8.x86_64 2022-11-28T01:15:44.247 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-8.5.0-17.el8.x86_64 2022-11-28T01:15:44.248 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.15-4.el8.x86_64 2022-11-28T01:15:44.248 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-threads-0.3.15-4.el8.x86_64 2022-11-28T01:15:44.248 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-11-28T01:15:44.248 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-11-28T01:15:44.248 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-11-28T01:15:44.249 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:44.249 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:15:44.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-rook 2022-11-28T01:15:44.982 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:44 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:15:46.144 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:15:46.145 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================= 2022-11-28T01:15:46.146 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2022-11-28T01:15:46.146 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================= 2022-11-28T01:15:46.146 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:15:46.146 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 47 k 2022-11-28T01:15:46.147 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2022-11-28T01:15:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-11-28T01:15:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-11-28T01:15:46.147 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-11-28T01:15:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-11-28T01:15:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-11-28T01:15:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-11-28T01:15:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2022-11-28T01:15:46.148 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-11-28T01:15:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-11-28T01:15:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2022-11-28T01:15:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-11-28T01:15:46.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-11-28T01:15:46.150 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:46.150 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:15:46.150 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================= 2022-11-28T01:15:46.150 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2022-11-28T01:15:46.151 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:46.151 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 1.9 M 2022-11-28T01:15:46.152 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 25 M 2022-11-28T01:15:46.152 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:15:46.203 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 494 kB/s | 27 kB 00:00 2022-11-28T01:15:46.237 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.5 MB/s | 126 kB 00:00 2022-11-28T01:15:46.254 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 192 kB/s | 20 kB 00:00 2022-11-28T01:15:46.279 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.6 MB/s | 110 kB 00:00 2022-11-28T01:15:46.304 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.0 MB/s | 155 kB 00:00 2022-11-28T01:15:46.321 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2022-11-28T01:15:46.430 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): ceph-mgr-rook-18.0.0-594.g697b0d99.el8. 167 kB/s | 47 kB 00:00 2022-11-28T01:15:46.455 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2022-11-28T01:15:46.481 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 187 kB/s | 33 kB 00:00 2022-11-28T01:15:46.531 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 807 kB/s | 81 kB 00:00 2022-11-28T01:15:46.581 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2022-11-28T01:15:46.615 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 460 kB/s | 61 kB 00:00 2022-11-28T01:15:46.723 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.4 MB/s | 1.2 MB 00:00 2022-11-28T01:15:46.725 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:15:46.726 INFO:teuthology.orchestra.run.smithi003.stdout:Total 3.3 MB/s | 1.9 MB 00:00 2022-11-28T01:15:46.778 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:15:46.794 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:15:46.794 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:15:46.873 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:15:46.877 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:15:47.378 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:15:47.663 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-11-28T01:15:47.836 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-11-28T01:15:48.061 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-11-28T01:15:48.282 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-11-28T01:15:48.490 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-11-28T01:15:48.733 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-11-28T01:15:49.037 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-11-28T01:15:49.794 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-11-28T01:15:50.912 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-11-28T01:15:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-11-28T01:15:51.365 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-11-28T01:15:51.543 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-11-28T01:15:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:18.0.0-594.g697b0d99.el8.noarch 13/13 2022-11-28T01:15:51.959 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-594.g697b0d99.el8.noarch 13/13 2022-11-28T01:15:51.959 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:18.0.0-594.g697b0d99.el8.noarch 1/13 2022-11-28T01:15:51.960 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-11-28T01:15:51.960 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-11-28T01:15:51.960 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-11-28T01:15:51.960 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-11-28T01:15:51.961 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-11-28T01:15:51.961 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-11-28T01:15:51.961 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-11-28T01:15:51.962 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-11-28T01:15:51.962 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-11-28T01:15:51.962 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-11-28T01:15:51.962 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-11-28T01:15:52.449 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-11-28T01:15:52.450 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:52.450 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:15:52.450 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-28T01:15:52.450 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-11-28T01:15:52.451 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-11-28T01:15:52.451 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-11-28T01:15:52.451 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-11-28T01:15:52.451 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-11-28T01:15:52.452 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-11-28T01:15:52.452 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-11-28T01:15:52.452 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-11-28T01:15:52.452 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-11-28T01:15:52.453 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-11-28T01:15:52.453 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.8-1.el8.noarch 2022-11-28T01:15:52.453 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-11-28T01:15:52.453 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:52.453 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:15:52.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-cephadm 2022-11-28T01:15:53.233 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:53 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:15:54.396 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:15:54.396 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:15:54.397 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2022-11-28T01:15:54.397 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:15:54.397 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:15:54.398 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 118 k 2022-11-28T01:15:54.398 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2022-11-28T01:15:54.398 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-11-28T01:15:54.398 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-11-28T01:15:54.399 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-11-28T01:15:54.399 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-11-28T01:15:54.399 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:54.399 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:15:54.400 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:15:54.400 INFO:teuthology.orchestra.run.smithi003.stdout:Install 5 Packages 2022-11-28T01:15:54.400 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:54.401 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 6.0 M 2022-11-28T01:15:54.401 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 26 M 2022-11-28T01:15:54.402 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:15:54.481 INFO:teuthology.orchestra.run.smithi003.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.5 MB/s | 538 kB 00:00 2022-11-28T01:15:54.615 INFO:teuthology.orchestra.run.smithi003.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 22 MB/s | 4.8 MB 00:00 2022-11-28T01:15:54.648 INFO:teuthology.orchestra.run.smithi003.stdout:(3/5): ceph-mgr-cephadm-18.0.0-594.g697b0d99.el 474 kB/s | 118 kB 00:00 2022-11-28T01:15:54.723 INFO:teuthology.orchestra.run.smithi003.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.9 MB/s | 476 kB 00:00 2022-11-28T01:15:54.740 INFO:teuthology.orchestra.run.smithi003.stdout:(5/5): python3-natsort-7.1.1-2.el8.noarch.rpm 482 kB/s | 60 kB 00:00 2022-11-28T01:15:54.742 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:15:54.743 INFO:teuthology.orchestra.run.smithi003.stdout:Total 17 MB/s | 6.0 MB 00:00 2022-11-28T01:15:54.795 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:15:54.807 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:15:54.807 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:15:54.944 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:15:54.947 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:15:55.261 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:15:55.579 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2022-11-28T01:15:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2022-11-28T01:15:57.070 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2022-11-28T01:15:57.294 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2022-11-28T01:15:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-594.g697b0d99.el8.noarch 5/5 2022-11-28T01:15:57.766 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-594.g697b0d99.el8.noarch 5/5 2022-11-28T01:15:57.767 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-594.g697b0d99.el8.noarch 1/5 2022-11-28T01:15:57.767 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2022-11-28T01:15:57.767 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2022-11-28T01:15:57.767 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2022-11-28T01:15:58.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2022-11-28T01:15:58.262 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:58.262 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:15:58.262 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-28T01:15:58.263 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-11-28T01:15:58.263 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.5.1-7.el8.noarch 2022-11-28T01:15:58.263 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-11-28T01:15:58.264 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-11-28T01:15:58.264 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:15:58.264 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:15:58.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse 2022-11-28T01:15:58.991 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:58 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:00.150 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:00.151 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:00.151 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2022-11-28T01:16:00.151 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:00.152 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:16:00.152 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:18.0.0-594.g697b0d99.el8 ceph 845 k 2022-11-28T01:16:00.152 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2022-11-28T01:16:00.152 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2022-11-28T01:16:00.153 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:00.153 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:16:00.153 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:00.153 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2022-11-28T01:16:00.154 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:00.154 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 928 k 2022-11-28T01:16:00.155 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.9 M 2022-11-28T01:16:00.155 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:16:00.254 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 824 kB/s | 83 kB 00:00 2022-11-28T01:16:00.464 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-fuse-18.0.0-594.g697b0d99.el8.x86_6 2.7 MB/s | 845 kB 00:00 2022-11-28T01:16:00.466 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:16:00.466 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.9 MB/s | 928 kB 00:00 2022-11-28T01:16:00.476 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:16:00.499 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:16:00.499 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:16:00.578 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:16:00.581 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:16:00.803 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:16:01.059 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2022-11-28T01:16:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 2/2 2022-11-28T01:16:01.594 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 2/2 2022-11-28T01:16:01.595 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 1/2 2022-11-28T01:16:02.068 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2022-11-28T01:16:02.068 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:02.068 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:16:02.069 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2022-11-28T01:16:02.069 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:02.069 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:02.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-volume 2022-11-28T01:16:02.906 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:02 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:04.059 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:04.060 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:04.061 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2022-11-28T01:16:04.061 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:04.061 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:16:04.061 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 263 k 2022-11-28T01:16:04.062 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:04.062 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:16:04.062 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:04.062 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2022-11-28T01:16:04.063 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:04.063 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 263 k 2022-11-28T01:16:04.063 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.3 M 2022-11-28T01:16:04.063 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:16:04.276 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-volume-18.0.0-594.g697b0d99.el8.noarch.rpm 1.2 MB/s | 263 kB 00:00 2022-11-28T01:16:04.277 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:16:04.278 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.2 MB/s | 263 kB 00:00 2022-11-28T01:16:04.278 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:16:04.288 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:16:04.289 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:16:04.306 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:16:04.309 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:16:04.483 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:16:04.755 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-volume-2:18.0.0-594.g697b0d99.el8.noarch 1/1 2022-11-28T01:16:05.132 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-volume-2:18.0.0-594.g697b0d99.el8.noarch 1/1 2022-11-28T01:16:05.793 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-volume-2:18.0.0-594.g697b0d99.el8.noarch 1/1 2022-11-28T01:16:05.793 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:05.794 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:16:05.794 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-28T01:16:05.794 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:05.794 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:05.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel 2022-11-28T01:16:06.623 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:06 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:07.776 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:07.777 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:07.777 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2022-11-28T01:16:07.777 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:07.778 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:16:07.778 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:18.0.0-594.g697b0d99.el8 ceph 139 k 2022-11-28T01:16:07.778 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:07.778 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:16:07.779 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:07.779 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2022-11-28T01:16:07.779 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:07.779 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 139 k 2022-11-28T01:16:07.780 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 512 k 2022-11-28T01:16:07.780 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:16:08.080 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-18.0.0-594.g697b0d99.el8.x86_64. 459 kB/s | 139 kB 00:00 2022-11-28T01:16:08.082 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:16:08.082 INFO:teuthology.orchestra.run.smithi003.stdout:Total 457 kB/s | 139 kB 00:00 2022-11-28T01:16:08.082 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:16:08.087 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:16:08.088 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:16:08.140 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:16:08.143 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:16:08.255 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:16:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:08.714 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:09.149 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:09.149 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:09.149 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:16:09.150 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:16:09.150 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:09.150 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:09.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2 2022-11-28T01:16:10.014 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:10 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:11.069 INFO:teuthology.orchestra.run.smithi003.stdout:Package libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 is already installed. 2022-11-28T01:16:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2022-11-28T01:16:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:11.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs-devel 2022-11-28T01:16:12.346 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:12 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:13.514 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:13.515 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:13.515 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2022-11-28T01:16:13.515 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:13.516 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:16:13.516 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:18.0.0-594.g697b0d99.el8 ceph 30 k 2022-11-28T01:16:13.516 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:13.516 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:16:13.516 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:13.517 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2022-11-28T01:16:13.517 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:13.517 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 30 k 2022-11-28T01:16:13.518 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 137 k 2022-11-28T01:16:13.518 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:16:13.671 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs-devel-18.0.0-594.g697b0d99.el8.x86_64 197 kB/s | 30 kB 00:00 2022-11-28T01:16:13.671 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:16:13.672 INFO:teuthology.orchestra.run.smithi003.stdout:Total 195 kB/s | 30 kB 00:00 2022-11-28T01:16:13.672 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:16:13.675 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:16:13.675 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:16:13.715 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:16:13.719 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:16:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:16:13.927 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:14.147 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:14.549 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:14.549 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:14.550 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:16:14.550 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:16:14.550 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:14.550 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:14.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2 2022-11-28T01:16:15.288 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:15 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:16.351 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:18.0.0-594.g697b0d99.el8.x86_64 is already installed. 2022-11-28T01:16:16.453 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:16.454 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2022-11-28T01:16:16.455 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:16.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1 2022-11-28T01:16:17.168 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:17 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:18.228 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 is already installed. 2022-11-28T01:16:18.331 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:18.332 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2022-11-28T01:16:18.333 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:18.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados 2022-11-28T01:16:19.042 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:19 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:20.097 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rados-2:18.0.0-594.g697b0d99.el8.x86_64 is already installed. 2022-11-28T01:16:20.201 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:20.203 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2022-11-28T01:16:20.203 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:20.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw 2022-11-28T01:16:20.926 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:20 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:21.989 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rgw-2:18.0.0-594.g697b0d99.el8.x86_64 is already installed. 2022-11-28T01:16:22.093 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:22.094 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2022-11-28T01:16:22.094 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:22.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs 2022-11-28T01:16:22.810 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:22 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:23.868 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-cephfs-2:18.0.0-594.g697b0d99.el8.x86_64 is already installed. 2022-11-28T01:16:23.971 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:23.973 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2022-11-28T01:16:23.973 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:24.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd 2022-11-28T01:16:24.687 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:24 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:25.741 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rbd-2:18.0.0-594.g697b0d99.el8.x86_64 is already installed. 2022-11-28T01:16:25.845 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:25.847 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2022-11-28T01:16:25.847 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:25.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse 2022-11-28T01:16:26.568 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:26 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:27.724 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:27.725 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:27.725 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2022-11-28T01:16:27.725 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:27.726 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:16:27.726 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:18.0.0-594.g697b0d99.el8 ceph 94 k 2022-11-28T01:16:27.726 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:27.726 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:16:27.727 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:27.727 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2022-11-28T01:16:27.727 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:27.727 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 94 k 2022-11-28T01:16:27.727 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 280 k 2022-11-28T01:16:27.728 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:16:27.908 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-18.0.0-594.g697b0d99.el8.x86_64.rpm 517 kB/s | 94 kB 00:00 2022-11-28T01:16:27.911 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:16:27.912 INFO:teuthology.orchestra.run.smithi003.stdout:Total 514 kB/s | 94 kB 00:00 2022-11-28T01:16:27.912 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:16:27.921 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:16:27.922 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:16:27.974 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:16:27.977 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:16:28.097 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:16:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:28.703 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:29.279 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:29.280 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:29.280 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:16:29.280 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:16:29.281 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:29.281 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:29.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror 2022-11-28T01:16:30.130 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:30 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:31.306 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:31.308 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:31.308 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2022-11-28T01:16:31.309 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:31.309 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:16:31.310 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:18.0.0-594.g697b0d99.el8 ceph 3.5 M 2022-11-28T01:16:31.310 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:31.311 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:16:31.311 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:31.312 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2022-11-28T01:16:31.312 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:31.313 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.5 M 2022-11-28T01:16:31.313 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 16 M 2022-11-28T01:16:31.314 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:16:31.748 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-18.0.0-594.g697b0d99.el8.x86_64.rpm 7.9 MB/s | 3.5 MB 00:00 2022-11-28T01:16:31.749 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:16:31.749 INFO:teuthology.orchestra.run.smithi003.stdout:Total 7.9 MB/s | 3.5 MB 00:00 2022-11-28T01:16:31.750 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:16:31.762 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:16:31.762 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:16:31.827 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:16:31.830 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:16:32.293 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:16:32.485 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:32.909 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:33.369 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:33.369 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:16:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:16:33.370 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:33.370 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:33.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd 2022-11-28T01:16:34.171 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:34 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:16:35.329 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:16:35.330 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:35.330 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2022-11-28T01:16:35.330 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:35.331 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:16:35.331 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:18.0.0-594.g697b0d99.el8 ceph 175 k 2022-11-28T01:16:35.331 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:35.331 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:16:35.332 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:16:35.332 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2022-11-28T01:16:35.332 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:35.332 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 175 k 2022-11-28T01:16:35.332 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 556 k 2022-11-28T01:16:35.333 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:16:35.524 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-18.0.0-594.g697b0d99.el8.x86_64.rpm 904 kB/s | 175 kB 00:00 2022-11-28T01:16:35.525 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:16:35.525 INFO:teuthology.orchestra.run.smithi003.stdout:Total 899 kB/s | 175 kB 00:00 2022-11-28T01:16:35.526 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:16:35.538 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:16:35.538 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:16:35.590 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:16:35.593 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:16:35.710 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:16:35.924 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:36.174 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:18.0.0-594.g697b0d99.el8.x86_64 1/1 2022-11-28T01:16:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:36.641 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:16:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-28T01:16:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:36.641 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:16:36.818 DEBUG:teuthology.parallel:result is None 2022-11-28T01:16:36.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo lsb_release -is 2022-11-28T01:16:36.858 INFO:teuthology.orchestra.run.smithi003.stdout:CentOSStream 2022-11-28T01:16:36.860 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-11-28T01:16:36.861 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=697b0d99d9a52df92b20c8e053dcd46c75f364a9 2022-11-28T01:16:42.023 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-11-28T01:16:42.051 INFO:teuthology.orchestra.run.smithi003.stdout:18.0.0-594.g697b0d99.el8 2022-11-28T01:16:42.053 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-594.g697b0d99.el8 2022-11-28T01:16:42.053 INFO:teuthology.task.install:The correct ceph version 18.0.0-594.g697b0d99 is installed. 2022-11-28T01:16:42.054 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-11-28T01:16:42.055 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:42.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-11-28T01:16:42.122 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-11-28T01:16:42.124 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:42.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2022-11-28T01:16:42.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-11-28T01:16:42.262 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-11-28T01:16:42.262 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:42.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2022-11-28T01:16:42.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-11-28T01:16:42.402 INFO:teuthology.run_tasks:Running task ceph... 2022-11-28T01:16:42.483 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-11-28T01:16:42.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /var/log/ceph 2022-11-28T01:16:42.513 INFO:tasks.ceph:Disabling ceph logrotate... 2022-11-28T01:16:42.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/logrotate.d/ceph 2022-11-28T01:16:42.583 INFO:tasks.ceph:Creating extra log directories... 2022-11-28T01:16:42.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-11-28T01:16:42.667 INFO:tasks.ceph:Creating ceph cluster ceph... 2022-11-28T01:16:42.668 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'setgroup': 'ceph', 'setuser': 'ceph', 'storage classes': 'LUKEWARM, FROZEN'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-11-28T01:16:42.668 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2022-11-23_17:40:47-rgw-main-distro-default-smithi/7089245', 'branch': 'main', 'description': 'rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '7089245', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2022-11-23_17:40:47-rgw-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 1, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'setgroup': 'ceph', 'setuser': 'ceph', 'storage classes': 'LUKEWARM, FROZEN'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9'}}, 'rgw': {'frontend': 'beast', 'storage classes': 'LUKEWARM, FROZEN'}, 's3tests': {'force-branch': 'ceph-master'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'osd.2', 'mgr.0', 'client.0']], 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_697b0d99d9a52df92b20c8e053dcd46c75f364a9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9', 'targets': {'smithi003.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCl6Xr36MljkuCgtHHJ/YjyJonBGGRiSMd8PwWDqax4BJXiLfifmPZV3p0oZyBwZvX/UwomH4SFk4j7ozMYG1BgG2L5u+/wrjnR77bUaNADcQ7V2hXnv4a7zE9h49woOThJra+237YGTt2WH8fuixZHe+JeRGnDgn10dNxO14NSYJygTaVcx9exe+sHFSf7g2QZRpHgS0ANaN2OFP7F9LqFkvJJGfoa5lLrn/KSIY1FUITr5rpx3fEQ6F29H2513mfk4hr95CEqPm6k3gL34+wa4/WpdFFtZZamsMuz5GN+76cCgT8gOz6rxSS3g1/18qoSokrmp4vMJVyzg2dG7YGpo8BsXHvFQjW3KG3fnvvmVpL+upCg2o5uJE9iD0nOZFfEHj+Z1Tn3OfgJjT1dVZyO9roluNxlozcKLoXa/N9w1epuspCccis2IFir4EaiHILYC9+xmS/bn6N2ruPHFAkW/KjHzeGiav1+HflQzoo6d8qYHcE7v5kGw3waPyiTThc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'rgw': ['client.0']}, {'s3tests': {'client.0': {'extra_attrs': ['lifecycle'], 'lc_debug_interval': 10, 'rgw_server': 'client.0', 'storage classes': 'LUKEWARM, FROZEN'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '4da97cf64e542f347ec47b7bdbe5eca99759f9b7', 'timestamp': '2022-11-23_17:40:47', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23126'} 2022-11-28T01:16:42.669 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-11-28T01:16:42.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/run/ceph 2022-11-28T01:16:42.786 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:42.787 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2022-11-28T01:16:42.845 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-11-28T01:16:42.846 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2022-11-28T01:16:42.904 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-11-28T01:16:42.904 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-11-28T01:16:42.905 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 381186 Links: 1 2022-11-28T01:16:42.905 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-11-28T01:16:42.905 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2022-11-28T01:16:42.905 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2022-11-28 01:16:32.688251495 +0000 2022-11-28T01:16:42.906 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2022-11-28 01:12:52.621395695 +0000 2022-11-28T01:16:42.906 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2022-11-28 01:12:52.621395695 +0000 2022-11-28T01:16:42.906 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2022-11-28T01:16:42.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2022-11-28T01:16:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2022-11-28T01:16:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2022-11-28T01:16:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000283022 s, 1.8 MB/s 2022-11-28T01:16:42.973 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2022-11-28T01:16:43.034 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2022-11-28T01:16:43.093 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-11-28T01:16:43.093 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-11-28T01:16:43.093 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 371500 Links: 1 2022-11-28T01:16:43.093 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-11-28T01:16:43.094 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2022-11-28T01:16:43.094 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2022-11-28 01:16:32.688251495 +0000 2022-11-28T01:16:43.094 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2022-11-28 01:12:53.196388217 +0000 2022-11-28T01:16:43.094 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2022-11-28 01:12:53.196388217 +0000 2022-11-28T01:16:43.095 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2022-11-28T01:16:43.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2022-11-28T01:16:43.160 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2022-11-28T01:16:43.160 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2022-11-28T01:16:43.160 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000208066 s, 2.5 MB/s 2022-11-28T01:16:43.161 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2022-11-28T01:16:43.220 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2022-11-28T01:16:43.278 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-11-28T01:16:43.279 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-11-28T01:16:43.279 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 371554 Links: 1 2022-11-28T01:16:43.279 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-11-28T01:16:43.280 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2022-11-28T01:16:43.280 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2022-11-28 01:16:32.688251495 +0000 2022-11-28T01:16:43.280 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2022-11-28 01:12:53.749381025 +0000 2022-11-28T01:16:43.281 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2022-11-28 01:12:53.749381025 +0000 2022-11-28T01:16:43.281 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2022-11-28T01:16:43.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2022-11-28T01:16:43.344 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2022-11-28T01:16:43.345 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2022-11-28T01:16:43.345 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000188837 s, 2.7 MB/s 2022-11-28T01:16:43.346 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2022-11-28T01:16:43.405 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2022-11-28T01:16:43.463 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-11-28T01:16:43.464 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-11-28T01:16:43.464 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 381327 Links: 1 2022-11-28T01:16:43.464 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-11-28T01:16:43.464 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2022-11-28T01:16:43.465 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2022-11-28 01:16:32.688251495 +0000 2022-11-28T01:16:43.465 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2022-11-28 01:12:54.319373612 +0000 2022-11-28T01:16:43.465 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2022-11-28 01:12:54.319373612 +0000 2022-11-28T01:16:43.465 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2022-11-28T01:16:43.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2022-11-28T01:16:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2022-11-28T01:16:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2022-11-28T01:16:43.530 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000219839 s, 2.3 MB/s 2022-11-28T01:16:43.532 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2022-11-28T01:16:43.591 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2022-11-28T01:16:43.592 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2022-11-28T01:16:43.592 INFO:tasks.ceph:Generating config... 2022-11-28T01:16:43.595 INFO:tasks.ceph:[client] debug rgw = 20 2022-11-28T01:16:43.596 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-11-28T01:16:43.597 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-11-28T01:16:43.597 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2022-11-28T01:16:43.598 INFO:tasks.ceph:[client] rgw lc debug interval = 10 2022-11-28T01:16:43.598 INFO:tasks.ceph:[client] setgroup = ceph 2022-11-28T01:16:43.599 INFO:tasks.ceph:[client] setuser = ceph 2022-11-28T01:16:43.600 INFO:tasks.ceph:[client] storage classes = LUKEWARM, FROZEN 2022-11-28T01:16:43.600 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-11-28T01:16:43.601 INFO:tasks.ceph:[mgr] debug ms = 1 2022-11-28T01:16:43.601 INFO:tasks.ceph:[mon] debug mon = 20 2022-11-28T01:16:43.602 INFO:tasks.ceph:[mon] debug ms = 1 2022-11-28T01:16:43.603 INFO:tasks.ceph:[mon] debug paxos = 20 2022-11-28T01:16:43.603 INFO:tasks.ceph:[osd] debug ms = 1 2022-11-28T01:16:43.604 INFO:tasks.ceph:[osd] debug osd = 20 2022-11-28T01:16:43.605 INFO:tasks.ceph:Setting up mon.a... 2022-11-28T01:16:43.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2022-11-28T01:16:43.681 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.keyring 2022-11-28T01:16:43.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2022-11-28T01:16:43.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-11-28T01:16:43.758 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.3')] 2022-11-28T01:16:43.759 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.3'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'setgroup': 'ceph', 'setuser': 'ceph', 'storage classes': 'LUKEWARM, FROZEN'}, 'mon.a': {}} 2022-11-28T01:16:43.760 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:43.761 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-11-28T01:16:43.819 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.3 --print /home/ubuntu/cephtest/ceph.monmap 2022-11-28T01:16:43.901 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:16:43.901 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:16:43.902 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2022-11-28T01:16:43.903 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: generated fsid b763e1e5-4156-4186-b9c8-f7e325a67a93 2022-11-28T01:16:43.903 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = pacific 2022-11-28T01:16:43.903 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 0 2022-11-28T01:16:43.904 INFO:teuthology.orchestra.run.smithi003.stdout:fsid b763e1e5-4156-4186-b9c8-f7e325a67a93 2022-11-28T01:16:43.904 INFO:teuthology.orchestra.run.smithi003.stdout:last_changed 2022-11-28T01:16:43.901023+0000 2022-11-28T01:16:43.904 INFO:teuthology.orchestra.run.smithi003.stdout:created 2022-11-28T01:16:43.901023+0000 2022-11-28T01:16:43.904 INFO:teuthology.orchestra.run.smithi003.stdout:min_mon_release 16 (pacific) 2022-11-28T01:16:43.905 INFO:teuthology.orchestra.run.smithi003.stdout:election_strategy: 1 2022-11-28T01:16:43.905 INFO:teuthology.orchestra.run.smithi003.stdout:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2022-11-28T01:16:43.905 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2022-11-28T01:16:43.906 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-11-28T01:16:43.961 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b763e1e5-4156-4186-b9c8-f7e325a67a93... 2022-11-28T01:16:43.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-11-28T01:16:44.047 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2022-11-28T01:16:44.047 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2022-11-28T01:16:44.048 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-11-28T01:16:44.048 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2022-11-28T01:16:44.048 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.048 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2022-11-28T01:16:44.049 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.049 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2022-11-28T01:16:44.049 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.049 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2022-11-28T01:16:44.049 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2022-11-28T01:16:44.050 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.050 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2022-11-28T01:16:44.050 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2022-11-28T01:16:44.050 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.051 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2022-11-28T01:16:44.051 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2022-11-28T01:16:44.051 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.051 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2022-11-28T01:16:44.051 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2022-11-28T01:16:44.052 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.052 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2022-11-28T01:16:44.052 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.052 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2022-11-28T01:16:44.053 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2022-11-28T01:16:44.053 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2022-11-28T01:16:44.053 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2022-11-28T01:16:44.053 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2022-11-28T01:16:44.053 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2022-11-28T01:16:44.054 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2022-11-28T01:16:44.054 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-11-28T01:16:44.054 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2022-11-28T01:16:44.054 INFO:teuthology.orchestra.run.smithi003.stdout: mon_allow_pool_size_one = true 2022-11-28T01:16:44.055 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.055 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-11-28T01:16:44.055 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.055 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2022-11-28T01:16:44.055 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.056 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2022-11-28T01:16:44.056 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.056 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2022-11-28T01:16:44.056 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2022-11-28T01:16:44.057 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2022-11-28T01:16:44.057 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = 172.21.15.3 2022-11-28T01:16:44.057 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = b763e1e5-4156-4186-b9c8-f7e325a67a93 2022-11-28T01:16:44.057 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.057 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2022-11-28T01:16:44.058 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2022-11-28T01:16:44.058 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.058 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2022-11-28T01:16:44.058 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2022-11-28T01:16:44.058 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.059 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2022-11-28T01:16:44.059 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2022-11-28T01:16:44.059 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.060 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2022-11-28T01:16:44.060 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2022-11-28T01:16:44.061 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2022-11-28T01:16:44.061 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.061 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2022-11-28T01:16:44.061 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2022-11-28T01:16:44.062 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.062 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2022-11-28T01:16:44.062 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.062 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2022-11-28T01:16:44.062 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.063 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2022-11-28T01:16:44.063 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.063 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2022-11-28T01:16:44.063 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2022-11-28T01:16:44.064 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.064 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2022-11-28T01:16:44.064 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2022-11-28T01:16:44.064 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-11-28T01:16:44.064 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2022-11-28T01:16:44.065 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.065 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2022-11-28T01:16:44.065 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-11-28T01:16:44.065 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2022-11-28T01:16:44.065 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2022-11-28T01:16:44.066 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2022-11-28T01:16:44.066 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2022-11-28T01:16:44.066 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2022-11-28T01:16:44.066 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2022-11-28T01:16:44.067 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.067 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2022-11-28T01:16:44.067 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-11-28T01:16:44.068 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2022-11-28T01:16:44.068 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2022-11-28T01:16:44.068 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2022-11-28T01:16:44.068 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2022-11-28T01:16:44.068 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2022-11-28T01:16:44.069 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2022-11-28T01:16:44.069 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2022-11-28T01:16:44.069 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2022-11-28T01:16:44.069 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2022-11-28T01:16:44.069 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.070 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-11-28T01:16:44.070 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2022-11-28T01:16:44.070 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2022-11-28T01:16:44.071 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.071 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2022-11-28T01:16:44.071 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-11-28T01:16:44.071 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-11-28T01:16:44.072 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.072 INFO:teuthology.orchestra.run.smithi003.stdout: # 1m isn't quite enough 2022-11-28T01:16:44.072 INFO:teuthology.orchestra.run.smithi003.stdout: mon_down_mkfs_grace = 2m 2022-11-28T01:16:44.072 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.073 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_filestore_osds = false 2022-11-28T01:16:44.073 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:44.073 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2022-11-28T01:16:44.073 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2022-11-28T01:16:44.073 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2022-11-28T01:16:44.074 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2022-11-28T01:16:44.074 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-11-28T01:16:44.074 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-11-28T01:16:44.074 INFO:teuthology.orchestra.run.smithi003.stdout: debug rgw = 20 2022-11-28T01:16:44.074 INFO:teuthology.orchestra.run.smithi003.stdout: rgw crypt require ssl = False 2022-11-28T01:16:44.075 INFO:teuthology.orchestra.run.smithi003.stdout: rgw crypt s3 kms backend = testing 2022-11-28T01:16:44.075 INFO:teuthology.orchestra.run.smithi003.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2022-11-28T01:16:44.075 INFO:teuthology.orchestra.run.smithi003.stdout: rgw lc debug interval = 10 2022-11-28T01:16:44.075 INFO:teuthology.orchestra.run.smithi003.stdout: setgroup = ceph 2022-11-28T01:16:44.076 INFO:teuthology.orchestra.run.smithi003.stdout: setuser = ceph 2022-11-28T01:16:44.076 INFO:teuthology.orchestra.run.smithi003.stdout: storage classes = "LUKEWARM, FROZEN" 2022-11-28T01:16:44.076 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2022-11-28T01:16:44.077 INFO:tasks.ceph:Creating admin key on mon.a... 2022-11-28T01:16:44.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2022-11-28T01:16:44.177 INFO:tasks.ceph:Copying monmap to all nodes... 2022-11-28T01:16:44.178 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:44.179 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2022-11-28T01:16:44.198 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:44.198 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2022-11-28T01:16:44.257 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi003.front.sepia.ceph.com 2022-11-28T01:16:44.257 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:44.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.keyring 2022-11-28T01:16:44.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-11-28T01:16:44.334 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:44.334 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-11-28T01:16:44.393 INFO:tasks.ceph:Setting up mon nodes... 2022-11-28T01:16:44.393 INFO:tasks.ceph:Setting up mgr nodes... 2022-11-28T01:16:44.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mgr/ceph-0 && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.0 /var/lib/ceph/mgr/ceph-0/keyring 2022-11-28T01:16:44.489 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mgr/ceph-0/keyring 2022-11-28T01:16:44.491 INFO:tasks.ceph:Setting up mds nodes... 2022-11-28T01:16:44.492 INFO:tasks.ceph_client:Setting up client nodes... 2022-11-28T01:16:44.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-11-28T01:16:44.538 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.client.0.keyring 2022-11-28T01:16:44.551 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-11-28T01:16:44.552 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2022-11-28T01:16:44.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2022-11-28T01:16:44.619 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2022-11-28T01:16:44.620 INFO:tasks.ceph:role: osd.0 2022-11-28T01:16:44.620 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com 2022-11-28T01:16:44.620 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-11-28T01:16:44.695 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-11-28T01:16:44.695 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2022-11-28T01:16:44.696 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-11-28T01:16:44.696 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-11-28T01:16:44.697 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-11-28T01:16:44.697 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2022-11-28T01:16:44.697 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-11-28T01:16:44.698 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-11-28T01:16:44.698 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-11-28T01:16:44.699 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-11-28T01:16:45.119 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2022-11-28T01:16:45.121 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2022-11-28T01:16:45.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2022-11-28T01:16:45.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2022-11-28T01:16:45.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2022-11-28T01:16:45.403 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2022-11-28T01:16:45.404 INFO:tasks.ceph:role: osd.1 2022-11-28T01:16:45.404 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com 2022-11-28T01:16:45.405 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-11-28T01:16:45.479 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-11-28T01:16:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2022-11-28T01:16:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-11-28T01:16:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-11-28T01:16:45.480 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-11-28T01:16:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2022-11-28T01:16:45.481 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-11-28T01:16:45.481 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-11-28T01:16:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-11-28T01:16:45.482 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-11-28T01:16:45.893 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2022-11-28T01:16:45.895 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2022-11-28T01:16:45.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2022-11-28T01:16:46.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2022-11-28T01:16:46.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2022-11-28T01:16:46.105 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2022-11-28T01:16:46.106 INFO:tasks.ceph:role: osd.2 2022-11-28T01:16:46.106 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com 2022-11-28T01:16:46.107 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-11-28T01:16:46.181 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-11-28T01:16:46.182 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2022-11-28T01:16:46.182 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-11-28T01:16:46.182 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-11-28T01:16:46.183 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-11-28T01:16:46.183 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2022-11-28T01:16:46.183 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-11-28T01:16:46.183 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-11-28T01:16:46.184 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-11-28T01:16:46.184 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-11-28T01:16:46.622 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2022-11-28T01:16:46.624 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2022-11-28T01:16:46.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2022-11-28T01:16:46.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2022-11-28T01:16:46.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-11-28T01:16:46.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:46.937+0000 7fa2f90215c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2022-11-28T01:16:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:46.939+0000 7fa2f90215c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2022-11-28T01:16:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:46.939+0000 7fa2f90215c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-11-28T01:16:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:46.939+0000 7fa2f90215c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-11-28T01:16:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:46.939+0000 7fa2f90215c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-11-28T01:16:46.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:46.944+0000 7fa2f90215c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2022-11-28T01:16:49.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-11-28T01:16:49.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-11-28T01:16:49.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:49.149+0000 7f02c07325c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2022-11-28T01:16:49.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:49.149+0000 7f02c07325c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2022-11-28T01:16:49.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:49.149+0000 7f02c07325c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-11-28T01:16:49.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:49.149+0000 7f02c07325c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-11-28T01:16:49.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:49.149+0000 7f02c07325c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-11-28T01:16:49.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:49.149+0000 7f02c07325c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2022-11-28T01:16:51.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-11-28T01:16:51.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-11-28T01:16:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:51.357+0000 7f46de3ea5c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2022-11-28T01:16:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:51.357+0000 7f46de3ea5c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2022-11-28T01:16:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:51.357+0000 7f46de3ea5c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-11-28T01:16:51.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:51.357+0000 7f46de3ea5c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-11-28T01:16:51.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:51.357+0000 7f46de3ea5c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-11-28T01:16:51.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:16:51.358+0000 7f46de3ea5c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2022-11-28T01:16:53.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-11-28T01:16:53.469 INFO:tasks.ceph:Reading keys from all nodes... 2022-11-28T01:16:53.470 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:53.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/mgr/ceph-0/keyring of=/dev/stdout 2022-11-28T01:16:53.541 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:53.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2022-11-28T01:16:53.612 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:53.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2022-11-28T01:16:53.680 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:53.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2022-11-28T01:16:53.750 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:53.750 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2022-11-28T01:16:53.809 INFO:tasks.ceph:Adding keys to all mons... 2022-11-28T01:16:53.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo tee -a /etc/ceph/ceph.keyring 2022-11-28T01:16:53.877 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.0] 2022-11-28T01:16:53.878 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQD8C4RjXhshHRAAFXKlu2Sh8FcfaUUofwWfvQ== 2022-11-28T01:16:53.878 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.0] 2022-11-28T01:16:53.879 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQD+C4RjD/HsNxAAd2SuZGLeYttmRNOh0TXFiA== 2022-11-28T01:16:53.879 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.1] 2022-11-28T01:16:53.879 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQABDIRjCa7vCBAAR3rLDJVLYwu4TXR4cgINpQ== 2022-11-28T01:16:53.879 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.2] 2022-11-28T01:16:53.880 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQADDIRjRKdaFRAAx5nOxof30JgrqfyXR7AV3w== 2022-11-28T01:16:53.880 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2022-11-28T01:16:53.880 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQD8C4RjRkUMIBAAaZnZfLBKDe9+4b6Z9Uwl0A== 2022-11-28T01:16:53.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.0 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-11-28T01:16:53.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-28T01:16:54.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-28T01:16:54.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-28T01:16:54.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-11-28T01:16:54.206 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-11-28T01:16:54.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2022-11-28T01:16:54.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-11-28T01:16:54.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2022-11-28T01:16:54.365 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-11-28T01:16:54.424 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2022-11-28T01:16:54.425 INFO:tasks.ceph.mon.a:Restarting daemon 2022-11-28T01:16:54.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2022-11-28T01:16:54.466 INFO:tasks.ceph.mon.a:Started 2022-11-28T01:16:54.467 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2022-11-28T01:16:54.467 INFO:tasks.ceph.mgr.0:Restarting daemon 2022-11-28T01:16:54.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i 0 2022-11-28T01:16:54.470 INFO:tasks.ceph.mgr.0:Started 2022-11-28T01:16:54.470 INFO:tasks.ceph:Setting crush tunables to default 2022-11-28T01:16:54.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd crush tunables default 2022-11-28T01:16:54.684 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2022-11-28T01:16:54.695 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2022-11-28T01:16:54.696 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:54.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2022-11-28T01:16:54.733 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:54.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2022-11-28T01:16:54.763 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:16:54.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2022-11-28T01:16:54.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new 08e9c1b7-5217-4527-9277-4a3e76f7f412 0 2022-11-28T01:16:54.971 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:54.968+0000 7f532ffae200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-11-28T01:16:55.100 INFO:teuthology.orchestra.run.smithi003.stdout:0 2022-11-28T01:16:55.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new 76aed755-6917-448b-836d-664421ae424a 1 2022-11-28T01:16:55.163 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:55.161+0000 7f532ffae200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-11-28T01:16:55.243 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:55.240+0000 7f532ffae200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-11-28T01:16:55.314 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:55.311+0000 7f532ffae200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-11-28T01:16:55.330 INFO:teuthology.orchestra.run.smithi003.stdout:1 2022-11-28T01:16:55.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new eeb27915-4b5e-4e0c-b8e6-75a572fd0950 2 2022-11-28T01:16:55.561 INFO:teuthology.orchestra.run.smithi003.stdout:2 2022-11-28T01:16:55.574 INFO:tasks.ceph.osd.0:Restarting daemon 2022-11-28T01:16:55.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2022-11-28T01:16:55.580 INFO:tasks.ceph.osd.0:Started 2022-11-28T01:16:55.580 INFO:tasks.ceph.osd.1:Restarting daemon 2022-11-28T01:16:55.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2022-11-28T01:16:55.583 INFO:tasks.ceph.osd.1:Started 2022-11-28T01:16:55.583 INFO:tasks.ceph.osd.2:Restarting daemon 2022-11-28T01:16:55.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2022-11-28T01:16:55.586 INFO:tasks.ceph.osd.2:Started 2022-11-28T01:16:55.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-11-28T01:16:55.858 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:16:55.858 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":5,"fsid":"b763e1e5-4156-4186-b9c8-f7e325a67a93","created":"2022-11-28T01:16:54.587105+0000","modified":"2022-11-28T01:16:55.558406+0000","last_up_change":"0.000000","last_in_change":"2022-11-28T01:16:55.558406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"08e9c1b7-5217-4527-9277-4a3e76f7f412","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76aed755-6917-448b-836d-664421ae424a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eeb27915-4b5e-4e0c-b8e6-75a572fd0950","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-11-28T01:16:55.870 INFO:tasks.ceph.ceph_manager.ceph:[] 2022-11-28T01:16:55.871 INFO:tasks.ceph:Waiting for OSDs to come up 2022-11-28T01:16:55.921 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:55.919+0000 7f532ffae200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-11-28T01:16:56.035 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:56.033+0000 7f532ffae200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-11-28T01:16:56.113 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:56.111+0000 7f532ffae200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-11-28T01:16:56.173 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-11-28T01:16:56.184 INFO:tasks.ceph.osd.0.smithi003.stderr:2022-11-28T01:16:56.182+0000 7fa00bd475c0 -1 Falling back to public interface 2022-11-28T01:16:56.192 INFO:tasks.ceph.osd.1.smithi003.stderr:2022-11-28T01:16:56.189+0000 7f7540d245c0 -1 Falling back to public interface 2022-11-28T01:16:56.251 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:56.249+0000 7f532ffae200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-11-28T01:16:56.276 INFO:tasks.ceph.osd.2.smithi003.stderr:2022-11-28T01:16:56.273+0000 7fac8e01a5c0 -1 Falling back to public interface 2022-11-28T01:16:56.354 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:56.352+0000 7f532ffae200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-11-28T01:16:56.367 INFO:teuthology.misc.health.smithi003.stdout: 2022-11-28T01:16:56.367 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":5,"fsid":"b763e1e5-4156-4186-b9c8-f7e325a67a93","created":"2022-11-28T01:16:54.587105+0000","modified":"2022-11-28T01:16:55.558406+0000","last_up_change":"0.000000","last_in_change":"2022-11-28T01:16:55.558406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"08e9c1b7-5217-4527-9277-4a3e76f7f412","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76aed755-6917-448b-836d-664421ae424a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eeb27915-4b5e-4e0c-b8e6-75a572fd0950","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-11-28T01:16:56.378 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-11-28T01:16:56.423 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:56.421+0000 7f532ffae200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-11-28T01:16:56.823 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:56.820+0000 7f532ffae200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-11-28T01:16:56.964 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:56.961+0000 7f532ffae200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-11-28T01:16:57.038 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:57.035+0000 7f532ffae200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-11-28T01:16:57.207 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:57.205+0000 7f532ffae200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-11-28T01:16:57.356 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:57.353+0000 7f532ffae200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-11-28T01:16:57.503 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:57.500+0000 7f532ffae200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-11-28T01:16:57.749 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:57.746+0000 7f532ffae200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-11-28T01:16:57.890 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:57.887+0000 7f532ffae200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-11-28T01:16:58.510 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:58.507+0000 7f532ffae200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-11-28T01:16:58.734 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:58.732+0000 7f532ffae200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-11-28T01:16:58.917 INFO:tasks.ceph.osd.1.smithi003.stderr:2022-11-28T01:16:58.914+0000 7f7540d245c0 -1 osd.1 0 log_to_monitors true 2022-11-28T01:16:58.944 INFO:tasks.ceph.osd.0.smithi003.stderr:2022-11-28T01:16:58.940+0000 7fa00bd475c0 -1 osd.0 0 log_to_monitors true 2022-11-28T01:16:58.978 INFO:tasks.ceph.osd.2.smithi003.stderr:2022-11-28T01:16:58.975+0000 7fac8e01a5c0 -1 osd.2 0 log_to_monitors true 2022-11-28T01:16:59.536 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:59.534+0000 7f532ffae200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-11-28T01:16:59.610 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:16:59.607+0000 7f532ffae200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-11-28T01:17:00.176 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:17:00.173+0000 7f532ffae200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-11-28T01:17:00.269 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:17:00.267+0000 7f532ffae200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-11-28T01:17:00.809 INFO:tasks.ceph.osd.0.smithi003.stderr:2022-11-28T01:17:00.807+0000 7f9fff8ee700 -1 osd.0 0 waiting for initial osdmap 2022-11-28T01:17:00.817 INFO:tasks.ceph.osd.1.smithi003.stderr:2022-11-28T01:17:00.815+0000 7f75348cb700 -1 osd.1 0 waiting for initial osdmap 2022-11-28T01:17:00.831 INFO:tasks.ceph.osd.0.smithi003.stderr:2022-11-28T01:17:00.830+0000 7f9ff7cc9700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-11-28T01:17:00.832 INFO:tasks.ceph.osd.1.smithi003.stderr:2022-11-28T01:17:00.830+0000 7f752cca6700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-11-28T01:17:01.356 INFO:tasks.ceph.mgr.0.smithi003.stderr:2022-11-28T01:17:01.353+0000 7f52edd14700 -1 mgr.server handle_report got status from non-daemon mon.a 2022-11-28T01:17:01.582 INFO:tasks.ceph.osd.2.smithi003.stderr:2022-11-28T01:17:01.580+0000 7fac81bc1700 -1 osd.2 0 waiting for initial osdmap 2022-11-28T01:17:01.596 INFO:tasks.ceph.osd.2.smithi003.stderr:2022-11-28T01:17:01.595+0000 7fac79f9c700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-11-28T01:17:02.682 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-11-28T01:17:03.010 INFO:teuthology.misc.health.smithi003.stdout: 2022-11-28T01:17:03.011 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":9,"fsid":"b763e1e5-4156-4186-b9c8-f7e325a67a93","created":"2022-11-28T01:16:54.587105+0000","modified":"2022-11-28T01:17:02.584762+0000","last_up_change":"2022-11-28T01:17:02.584762+0000","last_in_change":"2022-11-28T01:16:55.558406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-11-28T01:17:02.375638+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"08e9c1b7-5217-4527-9277-4a3e76f7f412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":106349},{"type":"v1","addr":"172.21.15.3:6801","nonce":106349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":106349},{"type":"v1","addr":"172.21.15.3:6803","nonce":106349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":106349},{"type":"v1","addr":"172.21.15.3:6807","nonce":106349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":106349},{"type":"v1","addr":"172.21.15.3:6805","nonce":106349}]},"public_addr":"172.21.15.3:6801/106349","cluster_addr":"172.21.15.3:6803/106349","heartbeat_back_addr":"172.21.15.3:6807/106349","heartbeat_front_addr":"172.21.15.3:6805/106349","state":["exists","up"]},{"osd":1,"uuid":"76aed755-6917-448b-836d-664421ae424a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":106351},{"type":"v1","addr":"172.21.15.3:6809","nonce":106351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":106351},{"type":"v1","addr":"172.21.15.3:6811","nonce":106351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":106351},{"type":"v1","addr":"172.21.15.3:6815","nonce":106351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":106351},{"type":"v1","addr":"172.21.15.3:6813","nonce":106351}]},"public_addr":"172.21.15.3:6809/106351","cluster_addr":"172.21.15.3:6811/106351","heartbeat_back_addr":"172.21.15.3:6815/106351","heartbeat_front_addr":"172.21.15.3:6813/106351","state":["exists","up"]},{"osd":2,"uuid":"eeb27915-4b5e-4e0c-b8e6-75a572fd0950","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":106380},{"type":"v1","addr":"172.21.15.3:6817","nonce":106380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":106380},{"type":"v1","addr":"172.21.15.3:6819","nonce":106380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":106380},{"type":"v1","addr":"172.21.15.3:6823","nonce":106380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":106380},{"type":"v1","addr":"172.21.15.3:6821","nonce":106380}]},"public_addr":"172.21.15.3:6817/106380","cluster_addr":"172.21.15.3:6819/106380","heartbeat_back_addr":"172.21.15.3:6823/106380","heartbeat_front_addr":"172.21.15.3:6821/106380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-28T01:16:59.912627+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-28T01:16:59.898210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-11-28T01:17:03.023 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-11-28T01:17:03.023 INFO:tasks.ceph:Creating RBD pool 2022-11-28T01:17:03.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd pool create rbd 8 2022-11-28T01:17:03.593 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'rbd' created 2022-11-28T01:17:03.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2022-11-28T01:17:04.595 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rbd' on pool 'rbd' 2022-11-28T01:17:04.607 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2022-11-28T01:17:04.608 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-11-28T01:17:04.609 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2022-11-28T01:17:04.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2022-11-28T01:17:04.610 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-11-28T01:17:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:05.004 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":5,"active_gid":4102,"active_name":"0","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":105939},{"type":"v1","addr":"172.21.15.3:6825","nonce":105939}]},"active_addr":"172.21.15.3:6825/105939","active_change":"2022-11-28T01:17:00.281685+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":4002549533}]},{"addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2861691255}]},{"addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":662414231}]},{"addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":4159915809}]}]}} 2022-11-28T01:17:05.010 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2022-11-28T01:17:05.011 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2022-11-28T01:17:05.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-11-28T01:17:05.335 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:05.335 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":11,"fsid":"b763e1e5-4156-4186-b9c8-f7e325a67a93","created":"2022-11-28T01:16:54.587105+0000","modified":"2022-11-28T01:17:04.586583+0000","last_up_change":"2022-11-28T01:17:02.584762+0000","last_in_change":"2022-11-28T01:16:55.558406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-11-28T01:17:02.375638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-11-28T01:17:03.371589+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"08e9c1b7-5217-4527-9277-4a3e76f7f412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":106349},{"type":"v1","addr":"172.21.15.3:6801","nonce":106349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":106349},{"type":"v1","addr":"172.21.15.3:6803","nonce":106349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":106349},{"type":"v1","addr":"172.21.15.3:6807","nonce":106349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":106349},{"type":"v1","addr":"172.21.15.3:6805","nonce":106349}]},"public_addr":"172.21.15.3:6801/106349","cluster_addr":"172.21.15.3:6803/106349","heartbeat_back_addr":"172.21.15.3:6807/106349","heartbeat_front_addr":"172.21.15.3:6805/106349","state":["exists","up"]},{"osd":1,"uuid":"76aed755-6917-448b-836d-664421ae424a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":106351},{"type":"v1","addr":"172.21.15.3:6809","nonce":106351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":106351},{"type":"v1","addr":"172.21.15.3:6811","nonce":106351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":106351},{"type":"v1","addr":"172.21.15.3:6815","nonce":106351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":106351},{"type":"v1","addr":"172.21.15.3:6813","nonce":106351}]},"public_addr":"172.21.15.3:6809/106351","cluster_addr":"172.21.15.3:6811/106351","heartbeat_back_addr":"172.21.15.3:6815/106351","heartbeat_front_addr":"172.21.15.3:6813/106351","state":["exists","up"]},{"osd":2,"uuid":"eeb27915-4b5e-4e0c-b8e6-75a572fd0950","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":106380},{"type":"v1","addr":"172.21.15.3:6817","nonce":106380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":106380},{"type":"v1","addr":"172.21.15.3:6819","nonce":106380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":106380},{"type":"v1","addr":"172.21.15.3:6823","nonce":106380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":106380},{"type":"v1","addr":"172.21.15.3:6821","nonce":106380}]},"public_addr":"172.21.15.3:6817/106380","cluster_addr":"172.21.15.3:6819/106380","heartbeat_back_addr":"172.21.15.3:6823/106380","heartbeat_front_addr":"172.21.15.3:6821/106380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-28T01:16:59.912627+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-28T01:16:59.898210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-28T01:16:59.994384+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-11-28T01:17:05.349 INFO:tasks.ceph.ceph_manager.ceph:all up! 2022-11-28T01:17:05.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-11-28T01:17:05.677 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:05.678 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":12,"fsid":"b763e1e5-4156-4186-b9c8-f7e325a67a93","created":"2022-11-28T01:16:54.587105+0000","modified":"2022-11-28T01:17:05.587252+0000","last_up_change":"2022-11-28T01:17:02.584762+0000","last_in_change":"2022-11-28T01:16:55.558406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-11-28T01:17:02.375638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-11-28T01:17:03.371589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"08e9c1b7-5217-4527-9277-4a3e76f7f412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":106349},{"type":"v1","addr":"172.21.15.3:6801","nonce":106349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":106349},{"type":"v1","addr":"172.21.15.3:6803","nonce":106349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":106349},{"type":"v1","addr":"172.21.15.3:6807","nonce":106349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":106349},{"type":"v1","addr":"172.21.15.3:6805","nonce":106349}]},"public_addr":"172.21.15.3:6801/106349","cluster_addr":"172.21.15.3:6803/106349","heartbeat_back_addr":"172.21.15.3:6807/106349","heartbeat_front_addr":"172.21.15.3:6805/106349","state":["exists","up"]},{"osd":1,"uuid":"76aed755-6917-448b-836d-664421ae424a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":106351},{"type":"v1","addr":"172.21.15.3:6809","nonce":106351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":106351},{"type":"v1","addr":"172.21.15.3:6811","nonce":106351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":106351},{"type":"v1","addr":"172.21.15.3:6815","nonce":106351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":106351},{"type":"v1","addr":"172.21.15.3:6813","nonce":106351}]},"public_addr":"172.21.15.3:6809/106351","cluster_addr":"172.21.15.3:6811/106351","heartbeat_back_addr":"172.21.15.3:6815/106351","heartbeat_front_addr":"172.21.15.3:6813/106351","state":["exists","up"]},{"osd":2,"uuid":"eeb27915-4b5e-4e0c-b8e6-75a572fd0950","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":106380},{"type":"v1","addr":"172.21.15.3:6817","nonce":106380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":106380},{"type":"v1","addr":"172.21.15.3:6819","nonce":106380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":106380},{"type":"v1","addr":"172.21.15.3:6823","nonce":106380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":106380},{"type":"v1","addr":"172.21.15.3:6821","nonce":106380}]},"public_addr":"172.21.15.3:6817/106380","cluster_addr":"172.21.15.3:6819/106380","heartbeat_back_addr":"172.21.15.3:6823/106380","heartbeat_front_addr":"172.21.15.3:6821/106380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-28T01:16:59.912627+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-28T01:16:59.898210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-28T01:16:59.994384+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-11-28T01:17:05.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2022-11-28T01:17:05.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2022-11-28T01:17:05.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2022-11-28T01:17:05.859 INFO:teuthology.orchestra.run.smithi003.stdout:34359738370 2022-11-28T01:17:05.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2022-11-28T01:17:05.880 INFO:teuthology.orchestra.run.smithi003.stdout:38654705668 2022-11-28T01:17:05.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-11-28T01:17:05.882 INFO:teuthology.orchestra.run.smithi003.stdout:34359738370 2022-11-28T01:17:05.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-11-28T01:17:06.199 INFO:teuthology.orchestra.run.smithi003.stdout:34359738369 2022-11-28T01:17:06.213 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.1 2022-11-28T01:17:06.268 INFO:teuthology.orchestra.run.smithi003.stdout:34359738369 2022-11-28T01:17:06.272 INFO:teuthology.orchestra.run.smithi003.stdout:38654705667 2022-11-28T01:17:06.284 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2022-11-28T01:17:06.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.2 2022-11-28T01:17:07.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2022-11-28T01:17:07.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-11-28T01:17:07.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-11-28T01:17:07.611 INFO:teuthology.orchestra.run.smithi003.stdout:34359738370 2022-11-28T01:17:07.630 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.1 2022-11-28T01:17:07.631 DEBUG:teuthology.parallel:result is None 2022-11-28T01:17:07.640 INFO:teuthology.orchestra.run.smithi003.stdout:38654705668 2022-11-28T01:17:07.654 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.2 2022-11-28T01:17:07.655 DEBUG:teuthology.parallel:result is None 2022-11-28T01:17:07.656 INFO:teuthology.orchestra.run.smithi003.stdout:34359738370 2022-11-28T01:17:07.670 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.0 2022-11-28T01:17:07.671 DEBUG:teuthology.parallel:result is None 2022-11-28T01:17:07.671 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2022-11-28T01:17:07.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-11-28T01:17:08.030 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:08.032 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2022-11-28T01:17:08.048 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2022-11-28T01:17:06.319673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25208,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":23360,"kb_avail":314547592,"statfs":{"total":322122547200,"available":322096734208,"internally_reserved":0,"allocated":1482752,"data_stored":1258418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23920640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.594033+0000","last_change":"2022-11-28T01:17:04.593085+0000","last_active":"2022-11-28T01:17:04.594033+0000","last_peered":"2022-11-28T01:17:04.594033+0000","last_clean":"2022-11-28T01:17:04.594033+0000","last_became_active":"2022-11-28T01:17:04.592796+0000","last_became_peered":"2022-11-28T01:17:04.592796+0000","last_unstale":"2022-11-28T01:17:04.594033+0000","last_undegraded":"2022-11-28T01:17:04.594033+0000","last_fullsized":"2022-11-28T01:17:04.594033+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T10:10:16.045402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.593571+0000","last_change":"2022-11-28T01:17:04.592331+0000","last_active":"2022-11-28T01:17:04.593571+0000","last_peered":"2022-11-28T01:17:04.593571+0000","last_clean":"2022-11-28T01:17:04.593571+0000","last_became_active":"2022-11-28T01:17:04.592004+0000","last_became_peered":"2022-11-28T01:17:04.592004+0000","last_unstale":"2022-11-28T01:17:04.593571+0000","last_undegraded":"2022-11-28T01:17:04.593571+0000","last_fullsized":"2022-11-28T01:17:04.593571+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T07:20:26.127423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.593518+0000","last_change":"2022-11-28T01:17:04.592247+0000","last_active":"2022-11-28T01:17:04.593518+0000","last_peered":"2022-11-28T01:17:04.593518+0000","last_clean":"2022-11-28T01:17:04.593518+0000","last_became_active":"2022-11-28T01:17:04.591835+0000","last_became_peered":"2022-11-28T01:17:04.591835+0000","last_unstale":"2022-11-28T01:17:04.593518+0000","last_undegraded":"2022-11-28T01:17:04.593518+0000","last_fullsized":"2022-11-28T01:17:04.593518+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T12:36:09.645723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-11-28T01:17:05.589775+0000","last_change":"2022-11-28T01:17:04.593633+0000","last_active":"2022-11-28T01:17:05.589775+0000","last_peered":"2022-11-28T01:17:05.589775+0000","last_clean":"2022-11-28T01:17:05.589775+0000","last_became_active":"2022-11-28T01:17:04.593381+0000","last_became_peered":"2022-11-28T01:17:04.593381+0000","last_unstale":"2022-11-28T01:17:05.589775+0000","last_undegraded":"2022-11-28T01:17:05.589775+0000","last_fullsized":"2022-11-28T01:17:05.589775+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T02:02:58.930244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.589287+0000","last_change":"2022-11-28T01:17:03.604271+0000","last_active":"2022-11-28T01:17:04.589287+0000","last_peered":"2022-11-28T01:17:04.589287+0000","last_clean":"2022-11-28T01:17:04.589287+0000","last_became_active":"2022-11-28T01:17:03.603473+0000","last_became_peered":"2022-11-28T01:17:03.603473+0000","last_unstale":"2022-11-28T01:17:04.589287+0000","last_undegraded":"2022-11-28T01:17:04.589287+0000","last_fullsized":"2022-11-28T01:17:04.589287+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:02.584762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:02.584762+0000","last_clean_scrub_stamp":"2022-11-28T01:17:02.584762+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T04:27:18.093056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.596326+0000","last_change":"2022-11-28T01:17:04.594767+0000","last_active":"2022-11-28T01:17:04.596326+0000","last_peered":"2022-11-28T01:17:04.596326+0000","last_clean":"2022-11-28T01:17:04.596326+0000","last_became_active":"2022-11-28T01:17:04.594489+0000","last_became_peered":"2022-11-28T01:17:04.594489+0000","last_unstale":"2022-11-28T01:17:04.596326+0000","last_undegraded":"2022-11-28T01:17:04.596326+0000","last_fullsized":"2022-11-28T01:17:04.596326+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T09:44:45.955933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-11-28T01:17:05.589775+0000","last_change":"2022-11-28T01:17:04.593606+0000","last_active":"2022-11-28T01:17:05.589775+0000","last_peered":"2022-11-28T01:17:05.589775+0000","last_clean":"2022-11-28T01:17:05.589775+0000","last_became_active":"2022-11-28T01:17:04.593310+0000","last_became_peered":"2022-11-28T01:17:04.593310+0000","last_unstale":"2022-11-28T01:17:05.589775+0000","last_undegraded":"2022-11-28T01:17:05.589775+0000","last_fullsized":"2022-11-28T01:17:05.589775+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T06:55:44.218401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.595422+0000","last_change":"2022-11-28T01:17:04.594328+0000","last_active":"2022-11-28T01:17:04.595422+0000","last_peered":"2022-11-28T01:17:04.595422+0000","last_clean":"2022-11-28T01:17:04.595422+0000","last_became_active":"2022-11-28T01:17:04.594079+0000","last_became_peered":"2022-11-28T01:17:04.594079+0000","last_unstale":"2022-11-28T01:17:04.595422+0000","last_undegraded":"2022-11-28T01:17:04.595422+0000","last_fullsized":"2022-11-28T01:17:04.595422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T04:48:54.636619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.593530+0000","last_change":"2022-11-28T01:17:04.592536+0000","last_active":"2022-11-28T01:17:04.593530+0000","last_peered":"2022-11-28T01:17:04.593530+0000","last_clean":"2022-11-28T01:17:04.593530+0000","last_became_active":"2022-11-28T01:17:04.591926+0000","last_became_peered":"2022-11-28T01:17:04.591926+0000","last_unstale":"2022-11-28T01:17:04.593530+0000","last_undegraded":"2022-11-28T01:17:04.593530+0000","last_fullsized":"2022-11-28T01:17:04.593530+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T02:53:10.896306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7592,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850008,"statfs":{"total":107374182400,"available":107366408192,"internally_reserved":0,"allocated":98304,"data_stored":25894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8808,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848792,"statfs":{"total":107374182400,"available":107365163008,"internally_reserved":0,"allocated":692224,"data_stored":616262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8808,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848792,"statfs":{"total":107374182400,"available":107365163008,"internally_reserved":0,"allocated":692224,"data_stored":616262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.215}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-11-28T01:17:08.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-11-28T01:17:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:08.370 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2022-11-28T01:17:08.381 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2022-11-28T01:17:08.319960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25208,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":23360,"kb_avail":314547592,"statfs":{"total":322122547200,"available":322096734208,"internally_reserved":0,"allocated":1482752,"data_stored":1258418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23920640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000286"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.594033+0000","last_change":"2022-11-28T01:17:04.593085+0000","last_active":"2022-11-28T01:17:04.594033+0000","last_peered":"2022-11-28T01:17:04.594033+0000","last_clean":"2022-11-28T01:17:04.594033+0000","last_became_active":"2022-11-28T01:17:04.592796+0000","last_became_peered":"2022-11-28T01:17:04.592796+0000","last_unstale":"2022-11-28T01:17:04.594033+0000","last_undegraded":"2022-11-28T01:17:04.594033+0000","last_fullsized":"2022-11-28T01:17:04.594033+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T10:10:16.045402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.593571+0000","last_change":"2022-11-28T01:17:04.592331+0000","last_active":"2022-11-28T01:17:04.593571+0000","last_peered":"2022-11-28T01:17:04.593571+0000","last_clean":"2022-11-28T01:17:04.593571+0000","last_became_active":"2022-11-28T01:17:04.592004+0000","last_became_peered":"2022-11-28T01:17:04.592004+0000","last_unstale":"2022-11-28T01:17:04.593571+0000","last_undegraded":"2022-11-28T01:17:04.593571+0000","last_fullsized":"2022-11-28T01:17:04.593571+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T07:20:26.127423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.593518+0000","last_change":"2022-11-28T01:17:04.592247+0000","last_active":"2022-11-28T01:17:04.593518+0000","last_peered":"2022-11-28T01:17:04.593518+0000","last_clean":"2022-11-28T01:17:04.593518+0000","last_became_active":"2022-11-28T01:17:04.591835+0000","last_became_peered":"2022-11-28T01:17:04.591835+0000","last_unstale":"2022-11-28T01:17:04.593518+0000","last_undegraded":"2022-11-28T01:17:04.593518+0000","last_fullsized":"2022-11-28T01:17:04.593518+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T12:36:09.645723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-11-28T01:17:05.589775+0000","last_change":"2022-11-28T01:17:04.593633+0000","last_active":"2022-11-28T01:17:05.589775+0000","last_peered":"2022-11-28T01:17:05.589775+0000","last_clean":"2022-11-28T01:17:05.589775+0000","last_became_active":"2022-11-28T01:17:04.593381+0000","last_became_peered":"2022-11-28T01:17:04.593381+0000","last_unstale":"2022-11-28T01:17:05.589775+0000","last_undegraded":"2022-11-28T01:17:05.589775+0000","last_fullsized":"2022-11-28T01:17:05.589775+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T02:02:58.930244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.589287+0000","last_change":"2022-11-28T01:17:03.604271+0000","last_active":"2022-11-28T01:17:04.589287+0000","last_peered":"2022-11-28T01:17:04.589287+0000","last_clean":"2022-11-28T01:17:04.589287+0000","last_became_active":"2022-11-28T01:17:03.603473+0000","last_became_peered":"2022-11-28T01:17:03.603473+0000","last_unstale":"2022-11-28T01:17:04.589287+0000","last_undegraded":"2022-11-28T01:17:04.589287+0000","last_fullsized":"2022-11-28T01:17:04.589287+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:02.584762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:02.584762+0000","last_clean_scrub_stamp":"2022-11-28T01:17:02.584762+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T04:27:18.093056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.596326+0000","last_change":"2022-11-28T01:17:04.594767+0000","last_active":"2022-11-28T01:17:04.596326+0000","last_peered":"2022-11-28T01:17:04.596326+0000","last_clean":"2022-11-28T01:17:04.596326+0000","last_became_active":"2022-11-28T01:17:04.594489+0000","last_became_peered":"2022-11-28T01:17:04.594489+0000","last_unstale":"2022-11-28T01:17:04.596326+0000","last_undegraded":"2022-11-28T01:17:04.596326+0000","last_fullsized":"2022-11-28T01:17:04.596326+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T09:44:45.955933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-11-28T01:17:05.589775+0000","last_change":"2022-11-28T01:17:04.593606+0000","last_active":"2022-11-28T01:17:05.589775+0000","last_peered":"2022-11-28T01:17:05.589775+0000","last_clean":"2022-11-28T01:17:05.589775+0000","last_became_active":"2022-11-28T01:17:04.593310+0000","last_became_peered":"2022-11-28T01:17:04.593310+0000","last_unstale":"2022-11-28T01:17:05.589775+0000","last_undegraded":"2022-11-28T01:17:05.589775+0000","last_fullsized":"2022-11-28T01:17:05.589775+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T06:55:44.218401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.595422+0000","last_change":"2022-11-28T01:17:04.594328+0000","last_active":"2022-11-28T01:17:04.595422+0000","last_peered":"2022-11-28T01:17:04.595422+0000","last_clean":"2022-11-28T01:17:04.595422+0000","last_became_active":"2022-11-28T01:17:04.594079+0000","last_became_peered":"2022-11-28T01:17:04.594079+0000","last_unstale":"2022-11-28T01:17:04.595422+0000","last_undegraded":"2022-11-28T01:17:04.595422+0000","last_fullsized":"2022-11-28T01:17:04.595422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T04:48:54.636619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-11-28T01:17:04.593530+0000","last_change":"2022-11-28T01:17:04.592536+0000","last_active":"2022-11-28T01:17:04.593530+0000","last_peered":"2022-11-28T01:17:04.593530+0000","last_clean":"2022-11-28T01:17:04.593530+0000","last_became_active":"2022-11-28T01:17:04.591926+0000","last_became_peered":"2022-11-28T01:17:04.591926+0000","last_unstale":"2022-11-28T01:17:04.593530+0000","last_undegraded":"2022-11-28T01:17:04.593530+0000","last_fullsized":"2022-11-28T01:17:04.593530+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-11-28T01:17:03.585723+0000","last_clean_scrub_stamp":"2022-11-28T01:17:03.585723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-11-29T02:53:10.896306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7592,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850008,"statfs":{"total":107374182400,"available":107366408192,"internally_reserved":0,"allocated":98304,"data_stored":25894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8808,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848792,"statfs":{"total":107374182400,"available":107365163008,"internally_reserved":0,"allocated":692224,"data_stored":616262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8808,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848792,"statfs":{"total":107374182400,"available":107365163008,"internally_reserved":0,"allocated":692224,"data_stored":616262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.215}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-11-28T01:17:08.383 INFO:tasks.ceph.ceph_manager.ceph:clean! 2022-11-28T01:17:08.383 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-11-28T01:17:08.383 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2022-11-28T01:17:08.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2022-11-28T01:17:08.748 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:08.749 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-11-28T01:17:08.762 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2022-11-28T01:17:08.763 INFO:teuthology.run_tasks:Running task rgw... 2022-11-28T01:17:08.783 DEBUG:tasks.rgw:config is {'client.0': None} 2022-11-28T01:17:08.783 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2022-11-28T01:17:08.784 INFO:tasks.rgw:Creating data pools 2022-11-28T01:17:08.784 DEBUG:tasks.rgw:Obtaining remote for client client.0 2022-11-28T01:17:08.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2022-11-28T01:17:10.022 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'default.rgw.buckets.data' created 2022-11-28T01:17:10.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2022-11-28T01:17:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-11-28T01:17:11.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2022-11-28T01:17:12.022 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'default.rgw.buckets.index' created 2022-11-28T01:17:12.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2022-11-28T01:17:13.043 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-11-28T01:17:13.061 DEBUG:tasks.rgw:Pools created 2022-11-28T01:17:13.062 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2022-11-28T01:17:13.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2022-11-28T01:17:13.063 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2022-11-28T01:17:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:13.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:13.148+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.057+0000 7ff64e61b780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:15.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.057+0000 7ff64e61b780 20 realm 2022-11-28T01:17:15.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.057+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.058+0000 7ff64e61b780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:15.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.058+0000 7ff64e61b780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:17:15.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.058+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.059+0000 7ff64e61b780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:15.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.059+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.059+0000 7ff64e61b780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:15.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.059+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.060+0000 7ff64e61b780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:15.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.060+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.060+0000 7ff64e61b780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:15.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.060+0000 7ff64e61b780 10 Creating default zonegroup 2022-11-28T01:17:15.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.061+0000 7ff64e61b780 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-11-28T01:17:15.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.071+0000 7ff64e61b780 10 WARNING: store->list_zones() returned r=-2 2022-11-28T01:17:15.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.071+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.072+0000 7ff64e61b780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:15.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.074+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.074+0000 7ff64e61b780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:15.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.074+0000 7ff64e61b780 10 could not read realm id: (2) No such file or directory 2022-11-28T01:17:15.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.074+0000 7ff64e61b780 10 WARNING: failed to set zone as default, r=-22 2022-11-28T01:17:15.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.074+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.074+0000 7ff64e61b780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:15.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.077+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.077+0000 7ff64e61b780 20 rados_obj.operate() r=0 bl.length=873 2022-11-28T01:17:15.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.077+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.078+0000 7ff64e61b780 20 rados_obj.operate() r=0 bl.length=398 2022-11-28T01:17:15.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.078+0000 7ff64e61b780 10 Using default name default 2022-11-28T01:17:15.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.078+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.081+0000 7ff64e61b780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:15.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.081+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.081+0000 7ff64e61b780 20 rados_obj.operate() r=0 bl.length=873 2022-11-28T01:17:15.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.081+0000 7ff64e61b780 20 zone default found 2022-11-28T01:17:15.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.081+0000 7ff64e61b780 4 Realm: () 2022-11-28T01:17:15.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.081+0000 7ff64e61b780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:17:15.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.081+0000 7ff64e61b780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:17:15.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.081+0000 7ff64e61b780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:17:15.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.081+0000 7ff64e61b780 20 zonegroup default 2022-11-28T01:17:15.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.081+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:15.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.081+0000 7ff64e61b780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:15.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:15.081+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:17.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:17.049+0000 7ff64e61b780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:17.049+0000 7ff64e61b780 20 rados->read ofs=0 len=0 2022-11-28T01:17:17.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:17.055+0000 7ff64e61b780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:17.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:17.055+0000 7ff64e61b780 20 started sync module instance, tier type = 2022-11-28T01:17:17.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:17.055+0000 7ff64e61b780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:17:19.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.081+0000 7ff64e61b780 20 add_watcher() i=0 2022-11-28T01:17:19.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.081+0000 7ff64e61b780 20 add_watcher() i=1 2022-11-28T01:17:19.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.081+0000 7ff64e61b780 20 add_watcher() i=2 2022-11-28T01:17:19.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.081+0000 7ff64e61b780 20 add_watcher() i=3 2022-11-28T01:17:19.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.081+0000 7ff64e61b780 20 add_watcher() i=4 2022-11-28T01:17:19.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.081+0000 7ff64e61b780 20 add_watcher() i=5 2022-11-28T01:17:19.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.081+0000 7ff64e61b780 20 add_watcher() i=6 2022-11-28T01:17:19.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.082+0000 7ff64e61b780 20 add_watcher() i=7 2022-11-28T01:17:19.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.082+0000 7ff64e61b780 2 all 8 watchers are set, enabling cache 2022-11-28T01:17:19.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.082+0000 7ff64e61b780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-11-28T01:17:19.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.082+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-11-28T01:17:19.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.082+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.082+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-11-28T01:17:19.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.082+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-11-28T01:17:19.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.082+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.083+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-11-28T01:17:19.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.083+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-11-28T01:17:19.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.083+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.083+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-11-28T01:17:19.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.083+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-11-28T01:17:19.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.083+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.083+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-11-28T01:17:19.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.083+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.4 2022-11-28T01:17:19.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.083+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.083+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.4 does not exist 2022-11-28T01:17:19.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.083+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.5 2022-11-28T01:17:19.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.083+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.084+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.5 does not exist 2022-11-28T01:17:19.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.084+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.6 2022-11-28T01:17:19.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.084+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.084+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.6 does not exist 2022-11-28T01:17:19.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.084+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.7 2022-11-28T01:17:19.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.084+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.084+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.7 does not exist 2022-11-28T01:17:19.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.084+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.8 2022-11-28T01:17:19.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.084+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.084+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.8 does not exist 2022-11-28T01:17:19.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.084+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.9 2022-11-28T01:17:19.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.084+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.9 does not exist 2022-11-28T01:17:19.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.10 2022-11-28T01:17:19.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.10 does not exist 2022-11-28T01:17:19.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.11 2022-11-28T01:17:19.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.11 does not exist 2022-11-28T01:17:19.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.12 2022-11-28T01:17:19.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.12 does not exist 2022-11-28T01:17:19.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.13 2022-11-28T01:17:19.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.13 does not exist 2022-11-28T01:17:19.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.14 2022-11-28T01:17:19.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.085+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.086+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.14 does not exist 2022-11-28T01:17:19.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.086+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.15 2022-11-28T01:17:19.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.086+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.086+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.15 does not exist 2022-11-28T01:17:19.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.086+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.16 2022-11-28T01:17:19.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.086+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.086+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.16 does not exist 2022-11-28T01:17:19.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.086+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.17 2022-11-28T01:17:19.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.086+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.086+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.17 does not exist 2022-11-28T01:17:19.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.086+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.18 2022-11-28T01:17:19.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.086+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.087+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.18 does not exist 2022-11-28T01:17:19.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.087+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.19 2022-11-28T01:17:19.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.087+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.087+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.19 does not exist 2022-11-28T01:17:19.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.087+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.20 2022-11-28T01:17:19.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.087+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.087+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.20 does not exist 2022-11-28T01:17:19.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.087+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.21 2022-11-28T01:17:19.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.087+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.087+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.21 does not exist 2022-11-28T01:17:19.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.087+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.22 2022-11-28T01:17:19.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.087+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.088+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.22 does not exist 2022-11-28T01:17:19.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.088+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.23 2022-11-28T01:17:19.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.088+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.088+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.23 does not exist 2022-11-28T01:17:19.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.088+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.24 2022-11-28T01:17:19.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.088+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.088+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.24 does not exist 2022-11-28T01:17:19.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.088+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.25 2022-11-28T01:17:19.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.088+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.088+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.25 does not exist 2022-11-28T01:17:19.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.088+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.26 2022-11-28T01:17:19.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.088+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.089+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.26 does not exist 2022-11-28T01:17:19.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.089+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.27 2022-11-28T01:17:19.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.089+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.089+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.27 does not exist 2022-11-28T01:17:19.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.089+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.28 2022-11-28T01:17:19.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.089+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.089+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.28 does not exist 2022-11-28T01:17:19.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.089+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.29 2022-11-28T01:17:19.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.089+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.089+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.29 does not exist 2022-11-28T01:17:19.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.089+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.30 2022-11-28T01:17:19.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.089+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.090+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.30 does not exist 2022-11-28T01:17:19.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.090+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.31 2022-11-28T01:17:19.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.090+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.090+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.31 does not exist 2022-11-28T01:17:19.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.090+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.32 2022-11-28T01:17:19.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.090+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.090+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.32 does not exist 2022-11-28T01:17:19.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.090+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.33 2022-11-28T01:17:19.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.090+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.090+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.33 does not exist 2022-11-28T01:17:19.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.090+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.34 2022-11-28T01:17:19.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.090+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.34 does not exist 2022-11-28T01:17:19.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.35 2022-11-28T01:17:19.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.35 does not exist 2022-11-28T01:17:19.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.36 2022-11-28T01:17:19.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.36 does not exist 2022-11-28T01:17:19.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.37 2022-11-28T01:17:19.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.37 does not exist 2022-11-28T01:17:19.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.38 2022-11-28T01:17:19.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.38 does not exist 2022-11-28T01:17:19.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.39 2022-11-28T01:17:19.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.091+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.092+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.39 does not exist 2022-11-28T01:17:19.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.092+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.40 2022-11-28T01:17:19.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.092+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.092+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.40 does not exist 2022-11-28T01:17:19.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.092+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.41 2022-11-28T01:17:19.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.092+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.092+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.41 does not exist 2022-11-28T01:17:19.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.092+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.42 2022-11-28T01:17:19.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.092+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.092+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.42 does not exist 2022-11-28T01:17:19.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.092+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.43 2022-11-28T01:17:19.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.092+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.093+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.43 does not exist 2022-11-28T01:17:19.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.093+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.44 2022-11-28T01:17:19.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.093+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.093+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.44 does not exist 2022-11-28T01:17:19.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.093+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.45 2022-11-28T01:17:19.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.093+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.093+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.45 does not exist 2022-11-28T01:17:19.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.093+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.46 2022-11-28T01:17:19.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.093+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.093+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.46 does not exist 2022-11-28T01:17:19.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.093+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.47 2022-11-28T01:17:19.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.093+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.47 does not exist 2022-11-28T01:17:19.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.48 2022-11-28T01:17:19.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.48 does not exist 2022-11-28T01:17:19.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.49 2022-11-28T01:17:19.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.49 does not exist 2022-11-28T01:17:19.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.50 2022-11-28T01:17:19.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.50 does not exist 2022-11-28T01:17:19.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.51 2022-11-28T01:17:19.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.51 does not exist 2022-11-28T01:17:19.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.52 2022-11-28T01:17:19.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.094+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.095+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.52 does not exist 2022-11-28T01:17:19.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.095+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.53 2022-11-28T01:17:19.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.095+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.095+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.53 does not exist 2022-11-28T01:17:19.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.095+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.54 2022-11-28T01:17:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.095+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.095+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.54 does not exist 2022-11-28T01:17:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.095+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.55 2022-11-28T01:17:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.095+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.095+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.55 does not exist 2022-11-28T01:17:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.095+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.56 2022-11-28T01:17:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.095+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.096+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.56 does not exist 2022-11-28T01:17:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.096+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.57 2022-11-28T01:17:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.096+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.096+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.57 does not exist 2022-11-28T01:17:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.096+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.58 2022-11-28T01:17:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.096+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.096+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.58 does not exist 2022-11-28T01:17:19.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.096+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.59 2022-11-28T01:17:19.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.096+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.096+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.59 does not exist 2022-11-28T01:17:19.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.096+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.60 2022-11-28T01:17:19.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.096+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.097+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.60 does not exist 2022-11-28T01:17:19.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.097+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.61 2022-11-28T01:17:19.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.097+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.097+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.61 does not exist 2022-11-28T01:17:19.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.097+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.62 2022-11-28T01:17:19.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.097+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.097+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.62 does not exist 2022-11-28T01:17:19.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.097+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.63 2022-11-28T01:17:19.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.097+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.097+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.63 does not exist 2022-11-28T01:17:19.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.097+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.64 2022-11-28T01:17:19.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.097+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.64 does not exist 2022-11-28T01:17:19.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.65 2022-11-28T01:17:19.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.65 does not exist 2022-11-28T01:17:19.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.66 2022-11-28T01:17:19.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.66 does not exist 2022-11-28T01:17:19.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.67 2022-11-28T01:17:19.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.67 does not exist 2022-11-28T01:17:19.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.68 2022-11-28T01:17:19.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.68 does not exist 2022-11-28T01:17:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.69 2022-11-28T01:17:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.098+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.099+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.69 does not exist 2022-11-28T01:17:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.099+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.70 2022-11-28T01:17:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.099+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.099+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.70 does not exist 2022-11-28T01:17:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.099+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.71 2022-11-28T01:17:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.099+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.099+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.71 does not exist 2022-11-28T01:17:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.099+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.72 2022-11-28T01:17:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.099+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.099+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.72 does not exist 2022-11-28T01:17:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.099+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.73 2022-11-28T01:17:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.099+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.100+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.73 does not exist 2022-11-28T01:17:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.100+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.74 2022-11-28T01:17:19.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.100+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.100+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.74 does not exist 2022-11-28T01:17:19.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.100+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.75 2022-11-28T01:17:19.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.100+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.100+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.75 does not exist 2022-11-28T01:17:19.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.100+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.76 2022-11-28T01:17:19.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.100+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.100+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.76 does not exist 2022-11-28T01:17:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.100+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.77 2022-11-28T01:17:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.100+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.77 does not exist 2022-11-28T01:17:19.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.78 2022-11-28T01:17:19.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.78 does not exist 2022-11-28T01:17:19.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.79 2022-11-28T01:17:19.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.79 does not exist 2022-11-28T01:17:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.80 2022-11-28T01:17:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.80 does not exist 2022-11-28T01:17:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.81 2022-11-28T01:17:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.81 does not exist 2022-11-28T01:17:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.82 2022-11-28T01:17:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.101+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.102+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.82 does not exist 2022-11-28T01:17:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.102+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.83 2022-11-28T01:17:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.102+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.102+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.83 does not exist 2022-11-28T01:17:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.102+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.84 2022-11-28T01:17:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.102+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.102+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.84 does not exist 2022-11-28T01:17:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.102+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.85 2022-11-28T01:17:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.102+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.102+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.85 does not exist 2022-11-28T01:17:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.102+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.86 2022-11-28T01:17:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.102+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.86 does not exist 2022-11-28T01:17:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.87 2022-11-28T01:17:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.87 does not exist 2022-11-28T01:17:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.88 2022-11-28T01:17:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.88 does not exist 2022-11-28T01:17:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.89 2022-11-28T01:17:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.89 does not exist 2022-11-28T01:17:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.90 2022-11-28T01:17:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.90 does not exist 2022-11-28T01:17:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.91 2022-11-28T01:17:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.103+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.104+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.91 does not exist 2022-11-28T01:17:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.104+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.92 2022-11-28T01:17:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.104+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.104+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.92 does not exist 2022-11-28T01:17:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.104+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.93 2022-11-28T01:17:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.104+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.104+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.93 does not exist 2022-11-28T01:17:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.104+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.94 2022-11-28T01:17:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.104+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.104+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.94 does not exist 2022-11-28T01:17:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.104+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.95 2022-11-28T01:17:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.104+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.95 does not exist 2022-11-28T01:17:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.96 2022-11-28T01:17:19.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.96 does not exist 2022-11-28T01:17:19.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.97 2022-11-28T01:17:19.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.97 does not exist 2022-11-28T01:17:19.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.98 2022-11-28T01:17:19.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.98 does not exist 2022-11-28T01:17:19.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.99 2022-11-28T01:17:19.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.99 does not exist 2022-11-28T01:17:19.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.100 2022-11-28T01:17:19.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.105+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.106+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.100 does not exist 2022-11-28T01:17:19.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.106+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.101 2022-11-28T01:17:19.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.106+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.106+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.101 does not exist 2022-11-28T01:17:19.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.106+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.102 2022-11-28T01:17:19.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.106+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.106+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.102 does not exist 2022-11-28T01:17:19.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.106+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.103 2022-11-28T01:17:19.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.106+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.106+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.103 does not exist 2022-11-28T01:17:19.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.106+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.104 2022-11-28T01:17:19.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.106+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.104 does not exist 2022-11-28T01:17:19.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.105 2022-11-28T01:17:19.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.105 does not exist 2022-11-28T01:17:19.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.106 2022-11-28T01:17:19.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.106 does not exist 2022-11-28T01:17:19.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.107 2022-11-28T01:17:19.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.107 does not exist 2022-11-28T01:17:19.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.108 2022-11-28T01:17:19.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.108 does not exist 2022-11-28T01:17:19.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.109 2022-11-28T01:17:19.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.107+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.108+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.109 does not exist 2022-11-28T01:17:19.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.108+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.110 2022-11-28T01:17:19.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.108+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.108+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.110 does not exist 2022-11-28T01:17:19.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.108+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.111 2022-11-28T01:17:19.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.108+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.108+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.111 does not exist 2022-11-28T01:17:19.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.108+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.112 2022-11-28T01:17:19.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.108+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.108+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.112 does not exist 2022-11-28T01:17:19.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.108+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.113 2022-11-28T01:17:19.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.108+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.113 does not exist 2022-11-28T01:17:19.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.114 2022-11-28T01:17:19.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.114 does not exist 2022-11-28T01:17:19.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.115 2022-11-28T01:17:19.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.115 does not exist 2022-11-28T01:17:19.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.116 2022-11-28T01:17:19.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.116 does not exist 2022-11-28T01:17:19.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.117 2022-11-28T01:17:19.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.117 does not exist 2022-11-28T01:17:19.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.118 2022-11-28T01:17:19.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.109+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.110+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.118 does not exist 2022-11-28T01:17:19.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.110+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.119 2022-11-28T01:17:19.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.110+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.110+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.119 does not exist 2022-11-28T01:17:19.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.110+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.120 2022-11-28T01:17:19.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.110+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.110+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.120 does not exist 2022-11-28T01:17:19.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.110+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.121 2022-11-28T01:17:19.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.110+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.110+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.121 does not exist 2022-11-28T01:17:19.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.110+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.122 2022-11-28T01:17:19.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.110+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.122 does not exist 2022-11-28T01:17:19.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.123 2022-11-28T01:17:19.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.123 does not exist 2022-11-28T01:17:19.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.124 2022-11-28T01:17:19.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.124 does not exist 2022-11-28T01:17:19.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.125 2022-11-28T01:17:19.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.125 does not exist 2022-11-28T01:17:19.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.126 2022-11-28T01:17:19.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.126 does not exist 2022-11-28T01:17:19.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.127 2022-11-28T01:17:19.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.111+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.112+0000 7ff64e61b780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.127 does not exist 2022-11-28T01:17:19.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.112+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-11-28T01:17:19.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.113+0000 7ff64e61b780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-11-28T01:17:19.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.116+0000 7ff633597700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:19.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.116+0000 7ff64e61b780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:19.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:19.116+0000 7ff64e61b780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:22.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.059+0000 7ff64e61b780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-11-28T01:17:22.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.059+0000 7ff64e61b780 5 note: GC not initialized 2022-11-28T01:17:22.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.061+0000 7ff64e61b780 20 init_complete bucket index max shards: 11 2022-11-28T01:17:22.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.061+0000 7ff64e61b780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:17:22.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.061+0000 7ff62cd8a700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:17:52 2022 2022-11-28T01:17:22.063 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:17:22.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.061+0000 7ff62c568700 20 reqs_thread_entry: start 2022-11-28T01:17:22.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.061+0000 7ff62ed8e700 20 reqs_thread_entry: start 2022-11-28T01:17:22.082 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2022-11-28T01:17:22.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.080+0000 7ff633597700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:22.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.083+0000 7ff64e61b780 20 remove_watcher() i=0 2022-11-28T01:17:22.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.083+0000 7ff64e61b780 2 removed watcher, disabling cache 2022-11-28T01:17:22.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.084+0000 7ff64e61b780 20 remove_watcher() i=1 2022-11-28T01:17:22.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.085+0000 7ff64e61b780 20 remove_watcher() i=2 2022-11-28T01:17:22.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.086+0000 7ff64e61b780 20 remove_watcher() i=3 2022-11-28T01:17:22.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.087+0000 7ff64e61b780 20 remove_watcher() i=4 2022-11-28T01:17:22.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.088+0000 7ff64e61b780 20 remove_watcher() i=5 2022-11-28T01:17:22.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.089+0000 7ff64e61b780 20 remove_watcher() i=6 2022-11-28T01:17:22.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.091+0000 7ff64e61b780 20 remove_watcher() i=7 2022-11-28T01:17:22.097 DEBUG:tasks.util.rgw: json result: [] 2022-11-28T01:17:22.097 INFO:tasks.rgw:Configuring storage class type = LUKEWARM 2022-11-28T01:17:22.098 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2022-11-28T01:17:22.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2022-11-28T01:17:22.099 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM 2022-11-28T01:17:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:22.155 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.163+0000 7f3c3583a780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.163+0000 7f3c3583a780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:22.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.167+0000 7f3c16faf700 20 reqs_thread_entry: start 2022-11-28T01:17:22.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.167+0000 7f3c3583a780 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-11-28T01:17:22.182 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-11-28T01:17:22.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:22.183 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "default-placement", 2022-11-28T01:17:22.183 INFO:teuthology.orchestra.run.smithi003.stdout: "val": { 2022-11-28T01:17:22.183 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default-placement", 2022-11-28T01:17:22.184 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2022-11-28T01:17:22.184 INFO:teuthology.orchestra.run.smithi003.stdout: "storage_classes": [ 2022-11-28T01:17:22.184 INFO:teuthology.orchestra.run.smithi003.stdout: "LUKEWARM", 2022-11-28T01:17:22.184 INFO:teuthology.orchestra.run.smithi003.stdout: "STANDARD" 2022-11-28T01:17:22.185 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-11-28T01:17:22.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:22.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:22.186 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-11-28T01:17:22.187 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['LUKEWARM', 'STANDARD']}}] 2022-11-28T01:17:22.187 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2022-11-28T01:17:22.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2022-11-28T01:17:22.188 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2022-11-28T01:17:22.242 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:22.242 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:22.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.249+0000 7fd9711c5780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:22.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.250+0000 7fd9711c5780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:22.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.253+0000 7fd95293a700 20 reqs_thread_entry: start 2022-11-28T01:17:22.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.253+0000 7fd9711c5780 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-11-28T01:17:22.268 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-11-28T01:17:22.268 INFO:teuthology.orchestra.run.smithi003.stdout: "id": "743c3e26-5a1f-4790-8db1-cbe73b4911f7", 2022-11-28T01:17:22.268 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default", 2022-11-28T01:17:22.269 INFO:teuthology.orchestra.run.smithi003.stdout: "domain_root": "default.rgw.meta:root", 2022-11-28T01:17:22.269 INFO:teuthology.orchestra.run.smithi003.stdout: "control_pool": "default.rgw.control", 2022-11-28T01:17:22.269 INFO:teuthology.orchestra.run.smithi003.stdout: "gc_pool": "default.rgw.log:gc", 2022-11-28T01:17:22.269 INFO:teuthology.orchestra.run.smithi003.stdout: "lc_pool": "default.rgw.log:lc", 2022-11-28T01:17:22.270 INFO:teuthology.orchestra.run.smithi003.stdout: "log_pool": "default.rgw.log", 2022-11-28T01:17:22.270 INFO:teuthology.orchestra.run.smithi003.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-11-28T01:17:22.270 INFO:teuthology.orchestra.run.smithi003.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-11-28T01:17:22.270 INFO:teuthology.orchestra.run.smithi003.stdout: "roles_pool": "default.rgw.meta:roles", 2022-11-28T01:17:22.270 INFO:teuthology.orchestra.run.smithi003.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-11-28T01:17:22.271 INFO:teuthology.orchestra.run.smithi003.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-11-28T01:17:22.271 INFO:teuthology.orchestra.run.smithi003.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-11-28T01:17:22.271 INFO:teuthology.orchestra.run.smithi003.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-11-28T01:17:22.271 INFO:teuthology.orchestra.run.smithi003.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-11-28T01:17:22.272 INFO:teuthology.orchestra.run.smithi003.stdout: "otp_pool": "default.rgw.otp", 2022-11-28T01:17:22.272 INFO:teuthology.orchestra.run.smithi003.stdout: "system_key": { 2022-11-28T01:17:22.272 INFO:teuthology.orchestra.run.smithi003.stdout: "access_key": "", 2022-11-28T01:17:22.272 INFO:teuthology.orchestra.run.smithi003.stdout: "secret_key": "" 2022-11-28T01:17:22.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:22.273 INFO:teuthology.orchestra.run.smithi003.stdout: "placement_pools": [ 2022-11-28T01:17:22.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:22.273 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "default-placement", 2022-11-28T01:17:22.273 INFO:teuthology.orchestra.run.smithi003.stdout: "val": { 2022-11-28T01:17:22.274 INFO:teuthology.orchestra.run.smithi003.stdout: "index_pool": "default.rgw.buckets.index", 2022-11-28T01:17:22.274 INFO:teuthology.orchestra.run.smithi003.stdout: "storage_classes": { 2022-11-28T01:17:22.274 INFO:teuthology.orchestra.run.smithi003.stdout: "LUKEWARM": { 2022-11-28T01:17:22.274 INFO:teuthology.orchestra.run.smithi003.stdout: "data_pool": "default.rgw.buckets.data.lukewarm" 2022-11-28T01:17:22.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:22.275 INFO:teuthology.orchestra.run.smithi003.stdout: "STANDARD": { 2022-11-28T01:17:22.275 INFO:teuthology.orchestra.run.smithi003.stdout: "data_pool": "default.rgw.buckets.data" 2022-11-28T01:17:22.275 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:22.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:22.276 INFO:teuthology.orchestra.run.smithi003.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-11-28T01:17:22.276 INFO:teuthology.orchestra.run.smithi003.stdout: "index_type": 0 2022-11-28T01:17:22.276 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:22.276 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:22.277 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:22.277 INFO:teuthology.orchestra.run.smithi003.stdout: "realm_id": "", 2022-11-28T01:17:22.277 INFO:teuthology.orchestra.run.smithi003.stdout: "notif_pool": "default.rgw.log:notif" 2022-11-28T01:17:22.277 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-11-28T01:17:22.279 DEBUG:tasks.util.rgw: json result: {'id': '743c3e26-5a1f-4790-8db1-cbe73b4911f7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-11-28T01:17:22.279 INFO:tasks.rgw:Configuring storage class type = FROZEN 2022-11-28T01:17:22.280 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2022-11-28T01:17:22.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2022-11-28T01:17:22.280 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN 2022-11-28T01:17:22.332 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:22.333 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:22.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.339+0000 7f2693e31780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:22.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.339+0000 7f2693e31780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:22.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.342+0000 7f26755a6700 20 reqs_thread_entry: start 2022-11-28T01:17:22.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.342+0000 7f2693e31780 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-11-28T01:17:22.357 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-11-28T01:17:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "default-placement", 2022-11-28T01:17:22.358 INFO:teuthology.orchestra.run.smithi003.stdout: "val": { 2022-11-28T01:17:22.358 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default-placement", 2022-11-28T01:17:22.358 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2022-11-28T01:17:22.358 INFO:teuthology.orchestra.run.smithi003.stdout: "storage_classes": [ 2022-11-28T01:17:22.359 INFO:teuthology.orchestra.run.smithi003.stdout: "FROZEN", 2022-11-28T01:17:22.359 INFO:teuthology.orchestra.run.smithi003.stdout: "LUKEWARM", 2022-11-28T01:17:22.359 INFO:teuthology.orchestra.run.smithi003.stdout: "STANDARD" 2022-11-28T01:17:22.359 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-11-28T01:17:22.359 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:22.360 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:22.360 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-11-28T01:17:22.361 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2022-11-28T01:17:22.361 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2022-11-28T01:17:22.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2022-11-28T01:17:22.362 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2022-11-28T01:17:22.415 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:22.416 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:22.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.423+0000 7ffa62e79780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:22.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.423+0000 7ffa62e79780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:22.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.426+0000 7ffa445ee700 20 reqs_thread_entry: start 2022-11-28T01:17:22.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:22.426+0000 7ffa62e79780 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-11-28T01:17:22.441 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-11-28T01:17:22.441 INFO:teuthology.orchestra.run.smithi003.stdout: "id": "743c3e26-5a1f-4790-8db1-cbe73b4911f7", 2022-11-28T01:17:22.441 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default", 2022-11-28T01:17:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: "domain_root": "default.rgw.meta:root", 2022-11-28T01:17:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: "control_pool": "default.rgw.control", 2022-11-28T01:17:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: "gc_pool": "default.rgw.log:gc", 2022-11-28T01:17:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: "lc_pool": "default.rgw.log:lc", 2022-11-28T01:17:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: "log_pool": "default.rgw.log", 2022-11-28T01:17:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-11-28T01:17:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-11-28T01:17:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: "roles_pool": "default.rgw.meta:roles", 2022-11-28T01:17:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-11-28T01:17:22.444 INFO:teuthology.orchestra.run.smithi003.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-11-28T01:17:22.444 INFO:teuthology.orchestra.run.smithi003.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-11-28T01:17:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-11-28T01:17:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-11-28T01:17:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: "otp_pool": "default.rgw.otp", 2022-11-28T01:17:22.446 INFO:teuthology.orchestra.run.smithi003.stdout: "system_key": { 2022-11-28T01:17:22.446 INFO:teuthology.orchestra.run.smithi003.stdout: "access_key": "", 2022-11-28T01:17:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: "secret_key": "" 2022-11-28T01:17:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: "placement_pools": [ 2022-11-28T01:17:22.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:22.448 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "default-placement", 2022-11-28T01:17:22.449 INFO:teuthology.orchestra.run.smithi003.stdout: "val": { 2022-11-28T01:17:22.449 INFO:teuthology.orchestra.run.smithi003.stdout: "index_pool": "default.rgw.buckets.index", 2022-11-28T01:17:22.450 INFO:teuthology.orchestra.run.smithi003.stdout: "storage_classes": { 2022-11-28T01:17:22.450 INFO:teuthology.orchestra.run.smithi003.stdout: "FROZEN": { 2022-11-28T01:17:22.450 INFO:teuthology.orchestra.run.smithi003.stdout: "data_pool": "default.rgw.buckets.data.frozen" 2022-11-28T01:17:22.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:22.451 INFO:teuthology.orchestra.run.smithi003.stdout: "LUKEWARM": { 2022-11-28T01:17:22.452 INFO:teuthology.orchestra.run.smithi003.stdout: "data_pool": "default.rgw.buckets.data.lukewarm" 2022-11-28T01:17:22.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:22.453 INFO:teuthology.orchestra.run.smithi003.stdout: "STANDARD": { 2022-11-28T01:17:22.453 INFO:teuthology.orchestra.run.smithi003.stdout: "data_pool": "default.rgw.buckets.data" 2022-11-28T01:17:22.453 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:22.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:22.454 INFO:teuthology.orchestra.run.smithi003.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-11-28T01:17:22.454 INFO:teuthology.orchestra.run.smithi003.stdout: "index_type": 0 2022-11-28T01:17:22.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:22.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:22.456 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:22.456 INFO:teuthology.orchestra.run.smithi003.stdout: "realm_id": "", 2022-11-28T01:17:22.456 INFO:teuthology.orchestra.run.smithi003.stdout: "notif_pool": "default.rgw.log:notif" 2022-11-28T01:17:22.457 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-11-28T01:17:22.458 DEBUG:tasks.util.rgw: json result: {'id': '743c3e26-5a1f-4790-8db1-cbe73b4911f7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-11-28T01:17:22.458 INFO:tasks.rgw:Starting rgw... 2022-11-28T01:17:22.459 INFO:tasks.rgw:rgw client.0 config is {} 2022-11-28T01:17:22.459 INFO:tasks.rgw:Using beast as radosgw frontend 2022-11-28T01:17:22.460 INFO:tasks.rgw.client.0:Restarting daemon 2022-11-28T01:17:22.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2022-11-28T01:17:22.463 INFO:tasks.rgw.client.0:Started 2022-11-28T01:17:22.463 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi003.front.sepia.ceph.com:80/ 2022-11-28T01:17:22.464 DEBUG:teuthology.orchestra.run.smithi003:> curl http://smithi003.front.sepia.ceph.com:80/ 2022-11-28T01:17:22.499 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-11-28T01:17:22.500 INFO:teuthology.orchestra.run.smithi003.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-11-28T01:17:22.500 INFO:teuthology.orchestra.run.smithi003.stderr: Dload Upload Total Spent Left Speed 2022-11-28T01:17:22.501 INFO:teuthology.orchestra.run.smithi003.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi003.front.sepia.ceph.com port 80: Connection refused 2022-11-28T01:17:23.502 DEBUG:teuthology.orchestra.run.smithi003:> curl http://smithi003.front.sepia.ceph.com:80/ 2022-11-28T01:17:23.526 INFO:teuthology.orchestra.run.smithi003.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-11-28T01:17:23.526 INFO:teuthology.orchestra.run.smithi003.stderr: Dload Upload Total Spent Left Speed 2022-11-28T01:17:23.528 INFO:teuthology.orchestra.run.smithi003.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2022-11-28T01:17:23.529 INFO:teuthology.orchestra.run.smithi003.stdout:anonymous 2022-11-28T01:17:23.529 INFO:teuthology.run_tasks:Running task s3tests... 2022-11-28T01:17:23.542 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'extra_attrs': ['lifecycle'], 'lc_debug_interval': 10, 'rgw_server': 'client.0', 'storage classes': 'LUKEWARM, FROZEN', 'force-branch': 'ceph-master'}} 2022-11-28T01:17:23.543 INFO:tasks.s3tests:Downloading s3-tests... 2022-11-28T01:17:23.544 INFO:tasks.s3tests:Using branch 'ceph-master' for s3tests 2022-11-28T01:17:23.544 DEBUG:teuthology.orchestra.run.smithi003:> git clone -b ceph-master https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests-client.0 2022-11-28T01:17:23.588 INFO:teuthology.orchestra.run.smithi003.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests-client.0'... 2022-11-28T01:17:23.935 INFO:tasks.s3tests:Creating rgw users... 2022-11-28T01:17:23.936 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2022-11-28T01:17:23.937 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --email foo.client.0+test@test.test --caps 'user-policy=*' --access-key XJYGHSERBPHNMPQLEAZF --secret l+48GsfiICVfzBDMo6/Sk0fMlJzDyz1PNHoJKtujnM4EI9EVc2nsyw== --cluster ceph 2022-11-28T01:17:24.033 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:24.033 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.040+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.041+0000 7fa31038a780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.041+0000 7fa31038a780 20 realm 2022-11-28T01:17:24.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.041+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.041+0000 7fa31038a780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.041+0000 7fa31038a780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:17:24.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.041+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.041+0000 7fa31038a780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.041+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.041+0000 7fa31038a780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:24.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.041+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.042+0000 7fa31038a780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:17:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.042+0000 7fa31038a780 20 searching for the correct realm 2022-11-28T01:17:24.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.049+0000 7fa31038a780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:17:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.049+0000 7fa31038a780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:17:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.049+0000 7fa31038a780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:17:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.049+0000 7fa31038a780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:17:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.049+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.049+0000 7fa31038a780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.049+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.049+0000 7fa31038a780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:17:24.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 20 zone default found 2022-11-28T01:17:24.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 4 Realm: () 2022-11-28T01:17:24.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:17:24.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:17:24.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:17:24.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 20 zonegroup default 2022-11-28T01:17:24.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.050+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.051+0000 7fa31038a780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.051+0000 7fa31038a780 20 started sync module instance, tier type = 2022-11-28T01:17:24.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.051+0000 7fa31038a780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:17:24.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.061+0000 7fa31038a780 20 add_watcher() i=0 2022-11-28T01:17:24.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.061+0000 7fa31038a780 20 add_watcher() i=1 2022-11-28T01:17:24.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.061+0000 7fa31038a780 20 add_watcher() i=2 2022-11-28T01:17:24.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.061+0000 7fa31038a780 20 add_watcher() i=3 2022-11-28T01:17:24.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.061+0000 7fa31038a780 20 add_watcher() i=4 2022-11-28T01:17:24.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.061+0000 7fa31038a780 20 add_watcher() i=5 2022-11-28T01:17:24.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.061+0000 7fa31038a780 20 add_watcher() i=6 2022-11-28T01:17:24.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.062+0000 7fa31038a780 20 add_watcher() i=7 2022-11-28T01:17:24.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.062+0000 7fa31038a780 2 all 8 watchers are set, enabling cache 2022-11-28T01:17:24.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.064+0000 7fa2f5306700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:24.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.064+0000 7fa31038a780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:24.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.064+0000 7fa31038a780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.068+0000 7fa31038a780 5 note: GC not initialized 2022-11-28T01:17:24.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.069+0000 7fa2f0afd700 20 reqs_thread_entry: start 2022-11-28T01:17:24.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.069+0000 7fa31038a780 20 init_complete bucket index max shards: 11 2022-11-28T01:17:24.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.069+0000 7fa31038a780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:17:24.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.069+0000 7fa2eeaf9700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:17:54 2022 2022-11-28T01:17:24.073 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:17:24.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.069+0000 7fa2ee2d7700 20 reqs_thread_entry: start 2022-11-28T01:17:24.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.069+0000 7fa31038a780 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2022-11-28T01:17:24.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.069+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2022-11-28T01:17:24.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2022-11-28T01:17:24.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2022-11-28T01:17:24.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2022-11-28T01:17:24.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2022-11-28T01:17:24.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 10 cache get: name=default.rgw.meta+users.keys+XJYGHSERBPHNMPQLEAZF : miss 2022-11-28T01:17:24.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 10 cache put: name=default.rgw.meta+users.keys+XJYGHSERBPHNMPQLEAZF info.flags=0x0 2022-11-28T01:17:24.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 10 adding default.rgw.meta+users.keys+XJYGHSERBPHNMPQLEAZF to cache LRU end 2022-11-28T01:17:24.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 10 cache get: name=default.rgw.meta+users.keys+XJYGHSERBPHNMPQLEAZF : hit (negative entry) 2022-11-28T01:17:24.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.070+0000 7fa31038a780 10 cache get: name=default.rgw.meta+users.keys+XJYGHSERBPHNMPQLEAZF : hit (negative entry) 2022-11-28T01:17:24.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.071+0000 7fa31038a780 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2022-11-28T01:17:24.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.071+0000 7fa31038a780 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2022-11-28T01:17:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.071+0000 7fa31038a780 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2022-11-28T01:17:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.071+0000 7fa308b2d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 94076404400000 notifier 4252 bl.length()=640 2022-11-28T01:17:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.071+0000 7fa308b2d700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2022-11-28T01:17:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.071+0000 7fa308b2d700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2022-11-28T01:17:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.073+0000 7fa31038a780 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2022-11-28T01:17:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.073+0000 7fa31038a780 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2022-11-28T01:17:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.073+0000 7fa31038a780 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2022-11-28T01:17:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.073+0000 7fa30832c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 94076404405760 notifier 4252 bl.length()=194 2022-11-28T01:17:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.073+0000 7fa30832c700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2022-11-28T01:17:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.073+0000 7fa30832c700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2022-11-28T01:17:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.074+0000 7fa31038a780 10 cache put: name=default.rgw.meta+users.keys+XJYGHSERBPHNMPQLEAZF info.flags=0x7 2022-11-28T01:17:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.074+0000 7fa31038a780 10 moving default.rgw.meta+users.keys+XJYGHSERBPHNMPQLEAZF to cache LRU end 2022-11-28T01:17:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.074+0000 7fa31038a780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:XJYGHSERBPHNMPQLEAZF, ofs0, ns] 2022-11-28T01:17:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.075+0000 7fa308b2d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182401 cookie 94076404404608 notifier 4252 bl.length()=186 2022-11-28T01:17:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.075+0000 7fa308b2d700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+XJYGHSERBPHNMPQLEAZF info.flags=0x7 2022-11-28T01:17:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.075+0000 7fa308b2d700 10 rgw watcher librados: moving default.rgw.meta+users.keys+XJYGHSERBPHNMPQLEAZF to cache LRU end 2022-11-28T01:17:24.083 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-11-28T01:17:24.083 INFO:teuthology.orchestra.run.smithi003.stdout: "user_id": "foo.client.0", 2022-11-28T01:17:24.084 INFO:teuthology.orchestra.run.smithi003.stdout: "display_name": "Mr. foo.client.0", 2022-11-28T01:17:24.084 INFO:teuthology.orchestra.run.smithi003.stdout: "email": "foo.client.0+test@test.test", 2022-11-28T01:17:24.084 INFO:teuthology.orchestra.run.smithi003.stdout: "suspended": 0, 2022-11-28T01:17:24.084 INFO:teuthology.orchestra.run.smithi003.stdout: "max_buckets": 1000, 2022-11-28T01:17:24.085 INFO:teuthology.orchestra.run.smithi003.stdout: "subusers": [], 2022-11-28T01:17:24.085 INFO:teuthology.orchestra.run.smithi003.stdout: "keys": [ 2022-11-28T01:17:24.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:24.085 INFO:teuthology.orchestra.run.smithi003.stdout: "user": "foo.client.0", 2022-11-28T01:17:24.086 INFO:teuthology.orchestra.run.smithi003.stdout: "access_key": "XJYGHSERBPHNMPQLEAZF", 2022-11-28T01:17:24.086 INFO:teuthology.orchestra.run.smithi003.stdout: "secret_key": "l+48GsfiICVfzBDMo6/Sk0fMlJzDyz1PNHoJKtujnM4EI9EVc2nsyw==" 2022-11-28T01:17:24.086 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:24.086 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:24.087 INFO:teuthology.orchestra.run.smithi003.stdout: "swift_keys": [], 2022-11-28T01:17:24.087 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": [ 2022-11-28T01:17:24.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:24.088 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "user-policy", 2022-11-28T01:17:24.088 INFO:teuthology.orchestra.run.smithi003.stdout: "perm": "*" 2022-11-28T01:17:24.089 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:24.089 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:24.090 INFO:teuthology.orchestra.run.smithi003.stdout: "op_mask": "read, write, delete", 2022-11-28T01:17:24.090 INFO:teuthology.orchestra.run.smithi003.stdout: "default_placement": "", 2022-11-28T01:17:24.091 INFO:teuthology.orchestra.run.smithi003.stdout: "default_storage_class": "", 2022-11-28T01:17:24.091 INFO:teuthology.orchestra.run.smithi003.stdout: "placement_tags": [], 2022-11-28T01:17:24.091 INFO:teuthology.orchestra.run.smithi003.stdout: "bucket_quota": { 2022-11-28T01:17:24.092 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": false, 2022-11-28T01:17:24.092 INFO:teuthology.orchestra.run.smithi003.stdout: "check_on_raw": false, 2022-11-28T01:17:24.092 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size": -1, 2022-11-28T01:17:24.093 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size_kb": 0, 2022-11-28T01:17:24.093 INFO:teuthology.orchestra.run.smithi003.stdout: "max_objects": -1 2022-11-28T01:17:24.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:24.094 INFO:teuthology.orchestra.run.smithi003.stdout: "user_quota": { 2022-11-28T01:17:24.094 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": false, 2022-11-28T01:17:24.095 INFO:teuthology.orchestra.run.smithi003.stdout: "check_on_raw": false, 2022-11-28T01:17:24.095 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size": -1, 2022-11-28T01:17:24.095 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size_kb": 0, 2022-11-28T01:17:24.095 INFO:teuthology.orchestra.run.smithi003.stdout: "max_objects": -1 2022-11-28T01:17:24.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:24.096 INFO:teuthology.orchestra.run.smithi003.stdout: "temp_url_keys": [], 2022-11-28T01:17:24.097 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "rgw", 2022-11-28T01:17:24.097 INFO:teuthology.orchestra.run.smithi003.stdout: "mfa_ids": [] 2022-11-28T01:17:24.097 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-11-28T01:17:24.098 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:24.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.076+0000 7fa2f5306700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:24.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.077+0000 7fa31038a780 20 remove_watcher() i=0 2022-11-28T01:17:24.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.077+0000 7fa31038a780 2 removed watcher, disabling cache 2022-11-28T01:17:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.078+0000 7fa31038a780 20 remove_watcher() i=1 2022-11-28T01:17:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.079+0000 7fa31038a780 20 remove_watcher() i=2 2022-11-28T01:17:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.080+0000 7fa31038a780 20 remove_watcher() i=3 2022-11-28T01:17:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.081+0000 7fa31038a780 20 remove_watcher() i=4 2022-11-28T01:17:24.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.082+0000 7fa31038a780 20 remove_watcher() i=5 2022-11-28T01:17:24.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.082+0000 7fa31038a780 20 remove_watcher() i=6 2022-11-28T01:17:24.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.083+0000 7fa31038a780 20 remove_watcher() i=7 2022-11-28T01:17:24.105 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 6785057245 --totp-seed IZ22GMHFWHRHJXQLNGT546XNDXJT7YZV6AJ4DK2FA2BCILSJUOC3O3TMI53ANSEZ --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2022-11-28T01:17:24.160 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:24.160 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:24.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.167+0000 7ffaf00b8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.167+0000 7ffaf00b8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.167+0000 7ffaf00b8780 20 realm 2022-11-28T01:17:24.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.167+0000 7ffaf00b8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.168+0000 7ffaf00b8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.168+0000 7ffaf00b8780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:17:24.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.168+0000 7ffaf00b8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.168+0000 7ffaf00b8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.168+0000 7ffaf00b8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.168+0000 7ffaf00b8780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:24.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.168+0000 7ffaf00b8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.169+0000 7ffaf00b8780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:17:24.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.169+0000 7ffaf00b8780 20 searching for the correct realm 2022-11-28T01:17:24.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.176+0000 7ffaf00b8780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:17:24.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.176+0000 7ffaf00b8780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:17:24.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.176+0000 7ffaf00b8780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:17:24.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.176+0000 7ffaf00b8780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:17:24.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.176+0000 7ffaf00b8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.176+0000 7ffaf00b8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.176+0000 7ffaf00b8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.176+0000 7ffaf00b8780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:24.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.176+0000 7ffaf00b8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:17:24.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 20 zone default found 2022-11-28T01:17:24.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 4 Realm: () 2022-11-28T01:17:24.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:17:24.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:17:24.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:17:24.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 20 zonegroup default 2022-11-28T01:17:24.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:24.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 20 started sync module instance, tier type = 2022-11-28T01:17:24.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.177+0000 7ffaf00b8780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:17:24.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.187+0000 7ffaf00b8780 20 add_watcher() i=0 2022-11-28T01:17:24.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.187+0000 7ffaf00b8780 20 add_watcher() i=1 2022-11-28T01:17:24.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.187+0000 7ffaf00b8780 20 add_watcher() i=2 2022-11-28T01:17:24.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.187+0000 7ffaf00b8780 20 add_watcher() i=3 2022-11-28T01:17:24.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.187+0000 7ffaf00b8780 20 add_watcher() i=4 2022-11-28T01:17:24.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.187+0000 7ffaf00b8780 20 add_watcher() i=5 2022-11-28T01:17:24.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.187+0000 7ffaf00b8780 20 add_watcher() i=6 2022-11-28T01:17:24.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.188+0000 7ffaf00b8780 20 add_watcher() i=7 2022-11-28T01:17:24.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.188+0000 7ffaf00b8780 2 all 8 watchers are set, enabling cache 2022-11-28T01:17:24.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.189+0000 7ffad5034700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:24.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.190+0000 7ffaf00b8780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:24.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.190+0000 7ffaf00b8780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:24.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.193+0000 7ffaf00b8780 5 note: GC not initialized 2022-11-28T01:17:24.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.194+0000 7ffad082b700 20 reqs_thread_entry: start 2022-11-28T01:17:24.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.194+0000 7ffaf00b8780 20 init_complete bucket index max shards: 11 2022-11-28T01:17:24.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.194+0000 7ffaf00b8780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:17:24.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.194+0000 7fface827700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:17:54 2022 2022-11-28T01:17:24.197 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:17:24.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.194+0000 7fface005700 20 reqs_thread_entry: start 2022-11-28T01:17:24.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.194+0000 7ffaf00b8780 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2022-11-28T01:17:24.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.194+0000 7ffaf00b8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:24.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.195+0000 7ffaf00b8780 20 rados_obj.operate() r=0 bl.length=455 2022-11-28T01:17:24.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.195+0000 7ffaf00b8780 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2022-11-28T01:17:24.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.195+0000 7ffaf00b8780 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2022-11-28T01:17:24.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:24.195+0000 7ffaf00b8780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2022-11-28T01:17:25.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.306+0000 7ffaf00b8780 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2022-11-28T01:17:25.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.306+0000 7ffaf00b8780 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2022-11-28T01:17:25.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.306+0000 7ffaf00b8780 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2022-11-28T01:17:25.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.307+0000 7ffae805a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 93986931711872 notifier 4260 bl.length()=654 2022-11-28T01:17:25.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.307+0000 7ffae805a700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2022-11-28T01:17:25.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.307+0000 7ffae805a700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2022-11-28T01:17:25.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.308+0000 7ffad5034700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:25.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.313+0000 7ffaf00b8780 20 remove_watcher() i=0 2022-11-28T01:17:25.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.313+0000 7ffaf00b8780 2 removed watcher, disabling cache 2022-11-28T01:17:25.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.314+0000 7ffaf00b8780 20 remove_watcher() i=1 2022-11-28T01:17:25.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.315+0000 7ffaf00b8780 20 remove_watcher() i=2 2022-11-28T01:17:25.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.316+0000 7ffaf00b8780 20 remove_watcher() i=3 2022-11-28T01:17:25.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.319+0000 7ffaf00b8780 20 remove_watcher() i=4 2022-11-28T01:17:25.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.320+0000 7ffaf00b8780 20 remove_watcher() i=5 2022-11-28T01:17:25.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.323+0000 7ffaf00b8780 20 remove_watcher() i=6 2022-11-28T01:17:25.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.324+0000 7ffaf00b8780 20 remove_watcher() i=7 2022-11-28T01:17:25.330 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2022-11-28T01:17:25.331 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --email bar.client.0+test@test.test --caps 'user-policy=*' --access-key FLQYKDHVOEAGUSRBWTRZ --secret +BsGcOhvzOcNZ3Fh4vjVqX6IOSYetyN+09FZoGXnihr+TQNRrJhQ1w== --cluster ceph 2022-11-28T01:17:25.390 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:25.391 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:25.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.399+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.399+0000 7fd176ed0780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.399+0000 7fd176ed0780 20 realm 2022-11-28T01:17:25.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.399+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.400+0000 7fd176ed0780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.400+0000 7fd176ed0780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:17:25.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.400+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.400+0000 7fd176ed0780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.400+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.400+0000 7fd176ed0780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:25.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.400+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.401+0000 7fd176ed0780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:17:25.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.401+0000 7fd176ed0780 20 searching for the correct realm 2022-11-28T01:17:25.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.408+0000 7fd176ed0780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:17:25.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.408+0000 7fd176ed0780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:17:25.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.408+0000 7fd176ed0780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:17:25.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.408+0000 7fd176ed0780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:17:25.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.408+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.408+0000 7fd176ed0780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.408+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:25.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:17:25.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 20 zone default found 2022-11-28T01:17:25.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 4 Realm: () 2022-11-28T01:17:25.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:17:25.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:17:25.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:17:25.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 20 zonegroup default 2022-11-28T01:17:25.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.409+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.410+0000 7fd176ed0780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.410+0000 7fd176ed0780 20 started sync module instance, tier type = 2022-11-28T01:17:25.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.410+0000 7fd176ed0780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:17:25.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.433+0000 7fd176ed0780 20 add_watcher() i=0 2022-11-28T01:17:25.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.433+0000 7fd176ed0780 20 add_watcher() i=1 2022-11-28T01:17:25.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.433+0000 7fd176ed0780 20 add_watcher() i=2 2022-11-28T01:17:25.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.433+0000 7fd176ed0780 20 add_watcher() i=3 2022-11-28T01:17:25.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.433+0000 7fd176ed0780 20 add_watcher() i=4 2022-11-28T01:17:25.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.433+0000 7fd176ed0780 20 add_watcher() i=5 2022-11-28T01:17:25.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.433+0000 7fd176ed0780 20 add_watcher() i=6 2022-11-28T01:17:25.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.433+0000 7fd176ed0780 20 add_watcher() i=7 2022-11-28T01:17:25.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.433+0000 7fd176ed0780 2 all 8 watchers are set, enabling cache 2022-11-28T01:17:25.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.435+0000 7fd15be4c700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:25.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.435+0000 7fd176ed0780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:25.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.435+0000 7fd176ed0780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:25.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.439+0000 7fd176ed0780 5 note: GC not initialized 2022-11-28T01:17:25.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.440+0000 7fd157643700 20 reqs_thread_entry: start 2022-11-28T01:17:25.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.440+0000 7fd176ed0780 20 init_complete bucket index max shards: 11 2022-11-28T01:17:25.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.440+0000 7fd176ed0780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:17:25.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.440+0000 7fd15563f700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:17:55 2022 2022-11-28T01:17:25.447 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:17:25.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.440+0000 7fd154e1d700 20 reqs_thread_entry: start 2022-11-28T01:17:25.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.440+0000 7fd176ed0780 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2022-11-28T01:17:25.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.440+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2022-11-28T01:17:25.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2022-11-28T01:17:25.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2022-11-28T01:17:25.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2022-11-28T01:17:25.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2022-11-28T01:17:25.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 10 cache get: name=default.rgw.meta+users.keys+FLQYKDHVOEAGUSRBWTRZ : miss 2022-11-28T01:17:25.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 10 cache put: name=default.rgw.meta+users.keys+FLQYKDHVOEAGUSRBWTRZ info.flags=0x0 2022-11-28T01:17:25.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 10 adding default.rgw.meta+users.keys+FLQYKDHVOEAGUSRBWTRZ to cache LRU end 2022-11-28T01:17:25.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 10 cache get: name=default.rgw.meta+users.keys+FLQYKDHVOEAGUSRBWTRZ : hit (negative entry) 2022-11-28T01:17:25.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.441+0000 7fd176ed0780 10 cache get: name=default.rgw.meta+users.keys+FLQYKDHVOEAGUSRBWTRZ : hit (negative entry) 2022-11-28T01:17:25.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.442+0000 7fd176ed0780 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2022-11-28T01:17:25.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.442+0000 7fd176ed0780 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2022-11-28T01:17:25.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.442+0000 7fd176ed0780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2022-11-28T01:17:25.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.443+0000 7fd16f673700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116993 cookie 94118295536640 notifier 4268 bl.length()=640 2022-11-28T01:17:25.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.443+0000 7fd16f673700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2022-11-28T01:17:25.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.443+0000 7fd16f673700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2022-11-28T01:17:25.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.444+0000 7fd176ed0780 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2022-11-28T01:17:25.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.444+0000 7fd176ed0780 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2022-11-28T01:17:25.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.444+0000 7fd176ed0780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2022-11-28T01:17:25.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.445+0000 7fd16ee72700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 94118295536640 notifier 4268 bl.length()=194 2022-11-28T01:17:25.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.445+0000 7fd16ee72700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2022-11-28T01:17:25.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.445+0000 7fd16ee72700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2022-11-28T01:17:25.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.446+0000 7fd176ed0780 10 cache put: name=default.rgw.meta+users.keys+FLQYKDHVOEAGUSRBWTRZ info.flags=0x7 2022-11-28T01:17:25.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.446+0000 7fd176ed0780 10 moving default.rgw.meta+users.keys+FLQYKDHVOEAGUSRBWTRZ to cache LRU end 2022-11-28T01:17:25.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.446+0000 7fd176ed0780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:FLQYKDHVOEAGUSRBWTRZ, ofs0, ns] 2022-11-28T01:17:25.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.446+0000 7fd16f673700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 94118295536640 notifier 4268 bl.length()=186 2022-11-28T01:17:25.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.446+0000 7fd16f673700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+FLQYKDHVOEAGUSRBWTRZ info.flags=0x7 2022-11-28T01:17:25.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.446+0000 7fd16f673700 10 rgw watcher librados: moving default.rgw.meta+users.keys+FLQYKDHVOEAGUSRBWTRZ to cache LRU end 2022-11-28T01:17:25.459 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-11-28T01:17:25.459 INFO:teuthology.orchestra.run.smithi003.stdout: "user_id": "bar.client.0", 2022-11-28T01:17:25.459 INFO:teuthology.orchestra.run.smithi003.stdout: "display_name": "Mr. bar.client.0", 2022-11-28T01:17:25.460 INFO:teuthology.orchestra.run.smithi003.stdout: "email": "bar.client.0+test@test.test", 2022-11-28T01:17:25.460 INFO:teuthology.orchestra.run.smithi003.stdout: "suspended": 0, 2022-11-28T01:17:25.460 INFO:teuthology.orchestra.run.smithi003.stdout: "max_buckets": 1000, 2022-11-28T01:17:25.461 INFO:teuthology.orchestra.run.smithi003.stdout: "subusers": [], 2022-11-28T01:17:25.461 INFO:teuthology.orchestra.run.smithi003.stdout: "keys": [ 2022-11-28T01:17:25.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:25.461 INFO:teuthology.orchestra.run.smithi003.stdout: "user": "bar.client.0", 2022-11-28T01:17:25.462 INFO:teuthology.orchestra.run.smithi003.stdout: "access_key": "FLQYKDHVOEAGUSRBWTRZ", 2022-11-28T01:17:25.462 INFO:teuthology.orchestra.run.smithi003.stdout: "secret_key": "+BsGcOhvzOcNZ3Fh4vjVqX6IOSYetyN+09FZoGXnihr+TQNRrJhQ1w==" 2022-11-28T01:17:25.463 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:25.463 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:25.463 INFO:teuthology.orchestra.run.smithi003.stdout: "swift_keys": [], 2022-11-28T01:17:25.463 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": [ 2022-11-28T01:17:25.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:25.464 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "user-policy", 2022-11-28T01:17:25.464 INFO:teuthology.orchestra.run.smithi003.stdout: "perm": "*" 2022-11-28T01:17:25.464 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:25.465 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:25.465 INFO:teuthology.orchestra.run.smithi003.stdout: "op_mask": "read, write, delete", 2022-11-28T01:17:25.465 INFO:teuthology.orchestra.run.smithi003.stdout: "default_placement": "", 2022-11-28T01:17:25.465 INFO:teuthology.orchestra.run.smithi003.stdout: "default_storage_class": "", 2022-11-28T01:17:25.466 INFO:teuthology.orchestra.run.smithi003.stdout: "placement_tags": [], 2022-11-28T01:17:25.466 INFO:teuthology.orchestra.run.smithi003.stdout: "bucket_quota": { 2022-11-28T01:17:25.466 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": false, 2022-11-28T01:17:25.466 INFO:teuthology.orchestra.run.smithi003.stdout: "check_on_raw": false, 2022-11-28T01:17:25.466 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size": -1, 2022-11-28T01:17:25.467 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size_kb": 0, 2022-11-28T01:17:25.467 INFO:teuthology.orchestra.run.smithi003.stdout: "max_objects": -1 2022-11-28T01:17:25.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:25.467 INFO:teuthology.orchestra.run.smithi003.stdout: "user_quota": { 2022-11-28T01:17:25.468 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": false, 2022-11-28T01:17:25.468 INFO:teuthology.orchestra.run.smithi003.stdout: "check_on_raw": false, 2022-11-28T01:17:25.468 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size": -1, 2022-11-28T01:17:25.468 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size_kb": 0, 2022-11-28T01:17:25.468 INFO:teuthology.orchestra.run.smithi003.stdout: "max_objects": -1 2022-11-28T01:17:25.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:25.469 INFO:teuthology.orchestra.run.smithi003.stdout: "temp_url_keys": [], 2022-11-28T01:17:25.469 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "rgw", 2022-11-28T01:17:25.469 INFO:teuthology.orchestra.run.smithi003.stdout: "mfa_ids": [] 2022-11-28T01:17:25.469 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-11-28T01:17:25.470 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.447+0000 7fd15be4c700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.449+0000 7fd176ed0780 20 remove_watcher() i=0 2022-11-28T01:17:25.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.449+0000 7fd176ed0780 2 removed watcher, disabling cache 2022-11-28T01:17:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.450+0000 7fd176ed0780 20 remove_watcher() i=1 2022-11-28T01:17:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.453+0000 7fd176ed0780 20 remove_watcher() i=2 2022-11-28T01:17:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.454+0000 7fd176ed0780 20 remove_watcher() i=3 2022-11-28T01:17:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.454+0000 7fd176ed0780 20 remove_watcher() i=4 2022-11-28T01:17:25.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.455+0000 7fd176ed0780 20 remove_watcher() i=5 2022-11-28T01:17:25.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.456+0000 7fd176ed0780 20 remove_watcher() i=6 2022-11-28T01:17:25.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.457+0000 7fd176ed0780 20 remove_watcher() i=7 2022-11-28T01:17:25.474 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 0782443418 --totp-seed LA6IYJCJROQGZ7J56PL7BQO6UO6VHKFFGEKUS4XAUWFLKPN5D6GWOKLLCZIDJMDZ --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2022-11-28T01:17:25.530 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:25.530 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:25.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.537+0000 7f89b7377780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.538+0000 7f89b7377780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.538+0000 7f89b7377780 20 realm 2022-11-28T01:17:25.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.538+0000 7f89b7377780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.538+0000 7f89b7377780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.538+0000 7f89b7377780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:17:25.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.538+0000 7f89b7377780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.538+0000 7f89b7377780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.538+0000 7f89b7377780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.538+0000 7f89b7377780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:25.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.538+0000 7f89b7377780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.539+0000 7f89b7377780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:17:25.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.539+0000 7f89b7377780 20 searching for the correct realm 2022-11-28T01:17:25.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.546+0000 7f89b7377780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:17:25.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.546+0000 7f89b7377780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:17:25.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.546+0000 7f89b7377780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:17:25.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.546+0000 7f89b7377780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:17:25.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.546+0000 7f89b7377780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.546+0000 7f89b7377780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.546+0000 7f89b7377780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:25.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:17:25.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 20 zone default found 2022-11-28T01:17:25.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 4 Realm: () 2022-11-28T01:17:25.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:17:25.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:17:25.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:17:25.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 20 zonegroup default 2022-11-28T01:17:25.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.547+0000 7f89b7377780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.548+0000 7f89b7377780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.548+0000 7f89b7377780 20 started sync module instance, tier type = 2022-11-28T01:17:25.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.548+0000 7f89b7377780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:17:25.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.557+0000 7f89b7377780 20 add_watcher() i=0 2022-11-28T01:17:25.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.557+0000 7f89b7377780 20 add_watcher() i=1 2022-11-28T01:17:25.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.557+0000 7f89b7377780 20 add_watcher() i=2 2022-11-28T01:17:25.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.557+0000 7f89b7377780 20 add_watcher() i=3 2022-11-28T01:17:25.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.557+0000 7f89b7377780 20 add_watcher() i=4 2022-11-28T01:17:25.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.557+0000 7f89b7377780 20 add_watcher() i=5 2022-11-28T01:17:25.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.557+0000 7f89b7377780 20 add_watcher() i=6 2022-11-28T01:17:25.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.558+0000 7f89b7377780 20 add_watcher() i=7 2022-11-28T01:17:25.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.558+0000 7f89b7377780 2 all 8 watchers are set, enabling cache 2022-11-28T01:17:25.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.559+0000 7f899c2f3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:25.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.559+0000 7f89b7377780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:25.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.559+0000 7f89b7377780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:25.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.563+0000 7f89b7377780 5 note: GC not initialized 2022-11-28T01:17:25.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.564+0000 7f8997aea700 20 reqs_thread_entry: start 2022-11-28T01:17:25.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.564+0000 7f89b7377780 20 init_complete bucket index max shards: 11 2022-11-28T01:17:25.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.564+0000 7f89b7377780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:17:25.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.564+0000 7f8995ae6700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:17:56 2022 2022-11-28T01:17:25.568 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:17:25.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.564+0000 7f89952c4700 20 reqs_thread_entry: start 2022-11-28T01:17:25.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.564+0000 7f89b7377780 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2022-11-28T01:17:25.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.564+0000 7f89b7377780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.564+0000 7f89b7377780 20 rados_obj.operate() r=0 bl.length=455 2022-11-28T01:17:25.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.564+0000 7f89b7377780 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2022-11-28T01:17:25.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.564+0000 7f89b7377780 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2022-11-28T01:17:25.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.564+0000 7f89b7377780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2022-11-28T01:17:25.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.567+0000 7f89b7377780 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2022-11-28T01:17:25.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.567+0000 7f89b7377780 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2022-11-28T01:17:25.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.567+0000 7f89b7377780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2022-11-28T01:17:25.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.567+0000 7f89afb1a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94251866350592 notifier 4276 bl.length()=654 2022-11-28T01:17:25.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.567+0000 7f89afb1a700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2022-11-28T01:17:25.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.567+0000 7f89afb1a700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2022-11-28T01:17:25.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.568+0000 7f899c2f3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:25.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.570+0000 7f89b7377780 20 remove_watcher() i=0 2022-11-28T01:17:25.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.570+0000 7f89b7377780 2 removed watcher, disabling cache 2022-11-28T01:17:25.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.573+0000 7f89b7377780 20 remove_watcher() i=1 2022-11-28T01:17:25.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.573+0000 7f89b7377780 20 remove_watcher() i=2 2022-11-28T01:17:25.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.574+0000 7f89b7377780 20 remove_watcher() i=3 2022-11-28T01:17:25.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.575+0000 7f89b7377780 20 remove_watcher() i=4 2022-11-28T01:17:25.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.578+0000 7f89b7377780 20 remove_watcher() i=5 2022-11-28T01:17:25.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.579+0000 7f89b7377780 20 remove_watcher() i=6 2022-11-28T01:17:25.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.580+0000 7f89b7377780 20 remove_watcher() i=7 2022-11-28T01:17:25.586 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2022-11-28T01:17:25.586 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --access-key TEOAHZBNNNEQOSSXMKJQ --secret LmbRddVMGC+/+pWUR4toAqs6sSUourcMrukloMeVPyrNk28PblPdAw== --cluster ceph 2022-11-28T01:17:25.642 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:25.643 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:25.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.650+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.650+0000 7f80d3e78780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.651+0000 7f80d3e78780 20 realm 2022-11-28T01:17:25.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.651+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.651+0000 7f80d3e78780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.651+0000 7f80d3e78780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:17:25.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.651+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.651+0000 7f80d3e78780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.651+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.651+0000 7f80d3e78780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:25.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.651+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.652+0000 7f80d3e78780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:17:25.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.652+0000 7f80d3e78780 20 searching for the correct realm 2022-11-28T01:17:25.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.659+0000 7f80d3e78780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:17:25.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.659+0000 7f80d3e78780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:17:25.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.659+0000 7f80d3e78780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:17:25.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.659+0000 7f80d3e78780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:17:25.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.659+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.659+0000 7f80d3e78780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.659+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.659+0000 7f80d3e78780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:25.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.659+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.659+0000 7f80d3e78780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:17:25.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 20 zone default found 2022-11-28T01:17:25.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 4 Realm: () 2022-11-28T01:17:25.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:17:25.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:17:25.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:17:25.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 20 zonegroup default 2022-11-28T01:17:25.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 20 started sync module instance, tier type = 2022-11-28T01:17:25.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.660+0000 7f80d3e78780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:17:25.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.679+0000 7f80d3e78780 20 add_watcher() i=0 2022-11-28T01:17:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.679+0000 7f80d3e78780 20 add_watcher() i=1 2022-11-28T01:17:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.679+0000 7f80d3e78780 20 add_watcher() i=2 2022-11-28T01:17:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.679+0000 7f80d3e78780 20 add_watcher() i=3 2022-11-28T01:17:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.679+0000 7f80d3e78780 20 add_watcher() i=4 2022-11-28T01:17:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.679+0000 7f80d3e78780 20 add_watcher() i=5 2022-11-28T01:17:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.679+0000 7f80d3e78780 20 add_watcher() i=6 2022-11-28T01:17:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.681+0000 7f80d3e78780 20 add_watcher() i=7 2022-11-28T01:17:25.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.681+0000 7f80d3e78780 2 all 8 watchers are set, enabling cache 2022-11-28T01:17:25.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.684+0000 7f80b8df4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:25.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.684+0000 7f80d3e78780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:25.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.684+0000 7f80d3e78780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.688+0000 7f80d3e78780 5 note: GC not initialized 2022-11-28T01:17:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.688+0000 7f80b45eb700 20 reqs_thread_entry: start 2022-11-28T01:17:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 20 init_complete bucket index max shards: 11 2022-11-28T01:17:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:17:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80b25e7700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:17:56 2022 2022-11-28T01:17:25.692 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:17:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80b1dc5700 20 reqs_thread_entry: start 2022-11-28T01:17:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2022-11-28T01:17:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2022-11-28T01:17:25.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2022-11-28T01:17:25.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2022-11-28T01:17:25.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2022-11-28T01:17:25.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2022-11-28T01:17:25.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 10 cache get: name=default.rgw.meta+users.keys+TEOAHZBNNNEQOSSXMKJQ : miss 2022-11-28T01:17:25.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.689+0000 7f80d3e78780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.690+0000 7f80d3e78780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.690+0000 7f80d3e78780 10 cache put: name=default.rgw.meta+users.keys+TEOAHZBNNNEQOSSXMKJQ info.flags=0x0 2022-11-28T01:17:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.690+0000 7f80d3e78780 10 adding default.rgw.meta+users.keys+TEOAHZBNNNEQOSSXMKJQ to cache LRU end 2022-11-28T01:17:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.690+0000 7f80d3e78780 10 cache get: name=default.rgw.meta+users.keys+TEOAHZBNNNEQOSSXMKJQ : hit (negative entry) 2022-11-28T01:17:25.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.690+0000 7f80d3e78780 10 cache get: name=default.rgw.meta+users.keys+TEOAHZBNNNEQOSSXMKJQ : hit (negative entry) 2022-11-28T01:17:25.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.691+0000 7f80d3e78780 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2022-11-28T01:17:25.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.691+0000 7f80d3e78780 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2022-11-28T01:17:25.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.691+0000 7f80d3e78780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2022-11-28T01:17:25.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.691+0000 7f80cc61b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 94549872619904 notifier 4284 bl.length()=714 2022-11-28T01:17:25.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.691+0000 7f80cc61b700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2022-11-28T01:17:25.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.691+0000 7f80cc61b700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2022-11-28T01:17:25.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.693+0000 7f80d3e78780 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2022-11-28T01:17:25.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.693+0000 7f80d3e78780 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2022-11-28T01:17:25.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.693+0000 7f80d3e78780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2022-11-28T01:17:25.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.694+0000 7f80cc61b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94549872622208 notifier 4284 bl.length()=224 2022-11-28T01:17:25.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.694+0000 7f80cc61b700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2022-11-28T01:17:25.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.694+0000 7f80cc61b700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2022-11-28T01:17:25.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.696+0000 7f80d3e78780 10 cache put: name=default.rgw.meta+users.keys+TEOAHZBNNNEQOSSXMKJQ info.flags=0x7 2022-11-28T01:17:25.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.696+0000 7f80d3e78780 10 moving default.rgw.meta+users.keys+TEOAHZBNNNEQOSSXMKJQ to cache LRU end 2022-11-28T01:17:25.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.696+0000 7f80d3e78780 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.keys:TEOAHZBNNNEQOSSXMKJQ, ofs0, ns] 2022-11-28T01:17:25.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.696+0000 7f80cc61b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116997 cookie 94549872621056 notifier 4284 bl.length()=201 2022-11-28T01:17:25.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.696+0000 7f80cc61b700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+TEOAHZBNNNEQOSSXMKJQ info.flags=0x7 2022-11-28T01:17:25.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.696+0000 7f80cc61b700 10 rgw watcher librados: moving default.rgw.meta+users.keys+TEOAHZBNNNEQOSSXMKJQ to cache LRU end 2022-11-28T01:17:25.703 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-11-28T01:17:25.704 INFO:teuthology.orchestra.run.smithi003.stdout: "user_id": "testx$tenanteduser.client.0", 2022-11-28T01:17:25.704 INFO:teuthology.orchestra.run.smithi003.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2022-11-28T01:17:25.704 INFO:teuthology.orchestra.run.smithi003.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2022-11-28T01:17:25.704 INFO:teuthology.orchestra.run.smithi003.stdout: "suspended": 0, 2022-11-28T01:17:25.705 INFO:teuthology.orchestra.run.smithi003.stdout: "max_buckets": 1000, 2022-11-28T01:17:25.705 INFO:teuthology.orchestra.run.smithi003.stdout: "subusers": [], 2022-11-28T01:17:25.705 INFO:teuthology.orchestra.run.smithi003.stdout: "keys": [ 2022-11-28T01:17:25.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:25.706 INFO:teuthology.orchestra.run.smithi003.stdout: "user": "testx$tenanteduser.client.0", 2022-11-28T01:17:25.706 INFO:teuthology.orchestra.run.smithi003.stdout: "access_key": "TEOAHZBNNNEQOSSXMKJQ", 2022-11-28T01:17:25.706 INFO:teuthology.orchestra.run.smithi003.stdout: "secret_key": "LmbRddVMGC+/+pWUR4toAqs6sSUourcMrukloMeVPyrNk28PblPdAw==" 2022-11-28T01:17:25.707 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:25.707 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:25.708 INFO:teuthology.orchestra.run.smithi003.stdout: "swift_keys": [], 2022-11-28T01:17:25.708 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": [ 2022-11-28T01:17:25.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:25.709 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "user-policy", 2022-11-28T01:17:25.709 INFO:teuthology.orchestra.run.smithi003.stdout: "perm": "*" 2022-11-28T01:17:25.710 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:25.710 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:25.711 INFO:teuthology.orchestra.run.smithi003.stdout: "op_mask": "read, write, delete", 2022-11-28T01:17:25.711 INFO:teuthology.orchestra.run.smithi003.stdout: "default_placement": "", 2022-11-28T01:17:25.712 INFO:teuthology.orchestra.run.smithi003.stdout: "default_storage_class": "", 2022-11-28T01:17:25.712 INFO:teuthology.orchestra.run.smithi003.stdout: "placement_tags": [], 2022-11-28T01:17:25.712 INFO:teuthology.orchestra.run.smithi003.stdout: "bucket_quota": { 2022-11-28T01:17:25.713 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": false, 2022-11-28T01:17:25.713 INFO:teuthology.orchestra.run.smithi003.stdout: "check_on_raw": false, 2022-11-28T01:17:25.714 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size": -1, 2022-11-28T01:17:25.714 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size_kb": 0, 2022-11-28T01:17:25.715 INFO:teuthology.orchestra.run.smithi003.stdout: "max_objects": -1 2022-11-28T01:17:25.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:25.716 INFO:teuthology.orchestra.run.smithi003.stdout: "user_quota": { 2022-11-28T01:17:25.716 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": false, 2022-11-28T01:17:25.716 INFO:teuthology.orchestra.run.smithi003.stdout: "check_on_raw": false, 2022-11-28T01:17:25.717 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size": -1, 2022-11-28T01:17:25.717 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size_kb": 0, 2022-11-28T01:17:25.717 INFO:teuthology.orchestra.run.smithi003.stdout: "max_objects": -1 2022-11-28T01:17:25.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:25.718 INFO:teuthology.orchestra.run.smithi003.stdout: "temp_url_keys": [], 2022-11-28T01:17:25.719 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "rgw", 2022-11-28T01:17:25.719 INFO:teuthology.orchestra.run.smithi003.stdout: "mfa_ids": [] 2022-11-28T01:17:25.719 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-11-28T01:17:25.720 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:25.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.697+0000 7f80b8df4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:25.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.701+0000 7f80d3e78780 20 remove_watcher() i=0 2022-11-28T01:17:25.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.701+0000 7f80d3e78780 2 removed watcher, disabling cache 2022-11-28T01:17:25.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.702+0000 7f80d3e78780 20 remove_watcher() i=1 2022-11-28T01:17:25.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.703+0000 7f80d3e78780 20 remove_watcher() i=2 2022-11-28T01:17:25.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.704+0000 7f80d3e78780 20 remove_watcher() i=3 2022-11-28T01:17:25.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.706+0000 7f80d3e78780 20 remove_watcher() i=4 2022-11-28T01:17:25.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.707+0000 7f80d3e78780 20 remove_watcher() i=5 2022-11-28T01:17:25.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.709+0000 7f80d3e78780 20 remove_watcher() i=6 2022-11-28T01:17:25.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.710+0000 7f80d3e78780 20 remove_watcher() i=7 2022-11-28T01:17:25.724 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 6206583518 --totp-seed A4GI3DTUN2YQPVWCDY35N566QQEM2TMK2V6IDAHMYIO6LP7SCQ6XSJZXCUCZ2JUL --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2022-11-28T01:17:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.788+0000 7f6572ada780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.788+0000 7f6572ada780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.788+0000 7f6572ada780 20 realm 2022-11-28T01:17:25.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.788+0000 7f6572ada780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.789+0000 7f6572ada780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.789+0000 7f6572ada780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:17:25.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.789+0000 7f6572ada780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.789+0000 7f6572ada780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.789+0000 7f6572ada780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.789+0000 7f6572ada780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.789+0000 7f6572ada780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.790+0000 7f6572ada780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:17:25.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.790+0000 7f6572ada780 20 searching for the correct realm 2022-11-28T01:17:25.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:17:25.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:17:25.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:17:25.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:17:25.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:25.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:17:25.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 zone default found 2022-11-28T01:17:25.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 4 Realm: () 2022-11-28T01:17:25.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:17:25.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:17:25.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:17:25.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 zonegroup default 2022-11-28T01:17:25.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.797+0000 7f6572ada780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.798+0000 7f6572ada780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.798+0000 7f6572ada780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.798+0000 7f6572ada780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.798+0000 7f6572ada780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.798+0000 7f6572ada780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.798+0000 7f6572ada780 20 started sync module instance, tier type = 2022-11-28T01:17:25.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.798+0000 7f6572ada780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:17:25.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.816+0000 7f6572ada780 20 add_watcher() i=0 2022-11-28T01:17:25.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.816+0000 7f6572ada780 20 add_watcher() i=1 2022-11-28T01:17:25.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.816+0000 7f6572ada780 20 add_watcher() i=2 2022-11-28T01:17:25.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.816+0000 7f6572ada780 20 add_watcher() i=3 2022-11-28T01:17:25.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.816+0000 7f6572ada780 20 add_watcher() i=4 2022-11-28T01:17:25.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.816+0000 7f6572ada780 20 add_watcher() i=5 2022-11-28T01:17:25.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.816+0000 7f6572ada780 20 add_watcher() i=6 2022-11-28T01:17:25.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.819+0000 7f6572ada780 20 add_watcher() i=7 2022-11-28T01:17:25.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.819+0000 7f6572ada780 2 all 8 watchers are set, enabling cache 2022-11-28T01:17:25.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.821+0000 7f6557a56700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:25.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.821+0000 7f6572ada780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:25.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.821+0000 7f6572ada780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:25.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.825+0000 7f6572ada780 5 note: GC not initialized 2022-11-28T01:17:25.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.826+0000 7f655324d700 20 reqs_thread_entry: start 2022-11-28T01:17:25.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.826+0000 7f6572ada780 20 init_complete bucket index max shards: 11 2022-11-28T01:17:25.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.826+0000 7f6572ada780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:17:25.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.826+0000 7f6551249700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:17:56 2022 2022-11-28T01:17:25.829 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:17:25.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.827+0000 7f6550a27700 20 reqs_thread_entry: start 2022-11-28T01:17:25.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.827+0000 7f6572ada780 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2022-11-28T01:17:25.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.827+0000 7f6572ada780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.827+0000 7f6572ada780 20 rados_obj.operate() r=0 bl.length=514 2022-11-28T01:17:25.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.827+0000 7f6572ada780 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2022-11-28T01:17:25.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.827+0000 7f6572ada780 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2022-11-28T01:17:25.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.827+0000 7f6572ada780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2022-11-28T01:17:25.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.831+0000 7f6572ada780 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2022-11-28T01:17:25.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.831+0000 7f6572ada780 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2022-11-28T01:17:25.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.831+0000 7f6572ada780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2022-11-28T01:17:25.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.831+0000 7f656b27d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116997 cookie 93847605588352 notifier 4292 bl.length()=728 2022-11-28T01:17:25.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.831+0000 7f656b27d700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2022-11-28T01:17:25.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.831+0000 7f656b27d700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2022-11-28T01:17:25.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.832+0000 7f6557a56700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:25.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.836+0000 7f6572ada780 20 remove_watcher() i=0 2022-11-28T01:17:25.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.836+0000 7f6572ada780 2 removed watcher, disabling cache 2022-11-28T01:17:25.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.837+0000 7f6572ada780 20 remove_watcher() i=1 2022-11-28T01:17:25.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.838+0000 7f6572ada780 20 remove_watcher() i=2 2022-11-28T01:17:25.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.839+0000 7f6572ada780 20 remove_watcher() i=3 2022-11-28T01:17:25.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.840+0000 7f6572ada780 20 remove_watcher() i=4 2022-11-28T01:17:25.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.841+0000 7f6572ada780 20 remove_watcher() i=5 2022-11-28T01:17:25.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.842+0000 7f6572ada780 20 remove_watcher() i=6 2022-11-28T01:17:25.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.843+0000 7f6572ada780 20 remove_watcher() i=7 2022-11-28T01:17:25.848 DEBUG:tasks.s3tests:Creating user foobar.client.0 on client.0 2022-11-28T01:17:25.849 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foobar.client.0 --display-name 'Mr. foobar.client.0' --email foobar.client.0+test@test.test --caps 'user-policy=*' --access-key FQXPRDMECGUAMFUROOWN --secret 3mO7d1HJbRSyuY3dzduv1pB/TPos68GWMP0ulZiW1LvKYsdF564MEA== --cluster ceph 2022-11-28T01:17:25.904 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:25.904 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:25.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.911+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.912+0000 7faf53619780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.912+0000 7faf53619780 20 realm 2022-11-28T01:17:25.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.912+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.912+0000 7faf53619780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.912+0000 7faf53619780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:17:25.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.912+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.912+0000 7faf53619780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.912+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.912+0000 7faf53619780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:25.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.912+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.913+0000 7faf53619780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:17:25.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.913+0000 7faf53619780 20 searching for the correct realm 2022-11-28T01:17:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:17:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:17:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:17:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:17:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:25.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:17:25.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 zone default found 2022-11-28T01:17:25.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 4 Realm: () 2022-11-28T01:17:25.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:17:25.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:17:25.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:17:25.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 zonegroup default 2022-11-28T01:17:25.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.921+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.922+0000 7faf53619780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.922+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.922+0000 7faf53619780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.922+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.922+0000 7faf53619780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.922+0000 7faf53619780 20 started sync module instance, tier type = 2022-11-28T01:17:25.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.922+0000 7faf53619780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:17:25.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.932+0000 7faf53619780 20 add_watcher() i=0 2022-11-28T01:17:25.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.932+0000 7faf53619780 20 add_watcher() i=1 2022-11-28T01:17:25.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.932+0000 7faf53619780 20 add_watcher() i=2 2022-11-28T01:17:25.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.932+0000 7faf53619780 20 add_watcher() i=3 2022-11-28T01:17:25.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.932+0000 7faf53619780 20 add_watcher() i=4 2022-11-28T01:17:25.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.932+0000 7faf53619780 20 add_watcher() i=5 2022-11-28T01:17:25.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.932+0000 7faf53619780 20 add_watcher() i=6 2022-11-28T01:17:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.933+0000 7faf53619780 20 add_watcher() i=7 2022-11-28T01:17:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.933+0000 7faf53619780 2 all 8 watchers are set, enabling cache 2022-11-28T01:17:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.935+0000 7faf38595700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.935+0000 7faf53619780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.935+0000 7faf53619780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.939+0000 7faf53619780 5 note: GC not initialized 2022-11-28T01:17:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.939+0000 7faf33d8c700 20 reqs_thread_entry: start 2022-11-28T01:17:25.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 20 init_complete bucket index max shards: 11 2022-11-28T01:17:25.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:17:25.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf31d88700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:17:56 2022 2022-11-28T01:17:25.943 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:17:25.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf31566700 20 reqs_thread_entry: start 2022-11-28T01:17:25.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2022-11-28T01:17:25.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x0 2022-11-28T01:17:25.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:17:25.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 10 cache get: name=default.rgw.meta+users.email+foobar.client.0+test@test.test : miss 2022-11-28T01:17:25.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 10 cache put: name=default.rgw.meta+users.email+foobar.client.0+test@test.test info.flags=0x0 2022-11-28T01:17:25.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 10 adding default.rgw.meta+users.email+foobar.client.0+test@test.test to cache LRU end 2022-11-28T01:17:25.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 10 cache get: name=default.rgw.meta+users.keys+FQXPRDMECGUAMFUROOWN : miss 2022-11-28T01:17:25.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.940+0000 7faf53619780 20 rados->read ofs=0 len=0 2022-11-28T01:17:25.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.941+0000 7faf53619780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:25.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.941+0000 7faf53619780 10 cache put: name=default.rgw.meta+users.keys+FQXPRDMECGUAMFUROOWN info.flags=0x0 2022-11-28T01:17:25.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.941+0000 7faf53619780 10 adding default.rgw.meta+users.keys+FQXPRDMECGUAMFUROOWN to cache LRU end 2022-11-28T01:17:25.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.941+0000 7faf53619780 10 cache get: name=default.rgw.meta+users.keys+FQXPRDMECGUAMFUROOWN : hit (negative entry) 2022-11-28T01:17:25.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.941+0000 7faf53619780 10 cache get: name=default.rgw.meta+users.keys+FQXPRDMECGUAMFUROOWN : hit (negative entry) 2022-11-28T01:17:25.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.942+0000 7faf53619780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2022-11-28T01:17:25.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.942+0000 7faf53619780 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:17:25.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.942+0000 7faf53619780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2022-11-28T01:17:25.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.942+0000 7faf4bdbc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116998 cookie 93854785469056 notifier 4300 bl.length()=655 2022-11-28T01:17:25.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.942+0000 7faf4bdbc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2022-11-28T01:17:25.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.942+0000 7faf4bdbc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:17:25.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.944+0000 7faf53619780 10 cache put: name=default.rgw.meta+users.email+foobar.client.0+test@test.test info.flags=0x7 2022-11-28T01:17:25.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.944+0000 7faf53619780 10 moving default.rgw.meta+users.email+foobar.client.0+test@test.test to cache LRU end 2022-11-28T01:17:25.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.944+0000 7faf53619780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.email:foobar.client.0+test@test.test, ofs0, ns] 2022-11-28T01:17:25.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.944+0000 7faf4b5bb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116999 cookie 93854785466752 notifier 4300 bl.length()=200 2022-11-28T01:17:25.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.944+0000 7faf4b5bb700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foobar.client.0+test@test.test info.flags=0x7 2022-11-28T01:17:25.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.944+0000 7faf4b5bb700 10 rgw watcher librados: moving default.rgw.meta+users.email+foobar.client.0+test@test.test to cache LRU end 2022-11-28T01:17:25.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.945+0000 7faf53619780 10 cache put: name=default.rgw.meta+users.keys+FQXPRDMECGUAMFUROOWN info.flags=0x7 2022-11-28T01:17:25.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.945+0000 7faf53619780 10 moving default.rgw.meta+users.keys+FQXPRDMECGUAMFUROOWN to cache LRU end 2022-11-28T01:17:25.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.945+0000 7faf53619780 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.keys:FQXPRDMECGUAMFUROOWN, ofs0, ns] 2022-11-28T01:17:25.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.946+0000 7faf4b5bb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116992 cookie 93854785464448 notifier 4300 bl.length()=189 2022-11-28T01:17:25.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.946+0000 7faf4b5bb700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+FQXPRDMECGUAMFUROOWN info.flags=0x7 2022-11-28T01:17:25.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.946+0000 7faf4b5bb700 10 rgw watcher librados: moving default.rgw.meta+users.keys+FQXPRDMECGUAMFUROOWN to cache LRU end 2022-11-28T01:17:25.955 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-11-28T01:17:25.956 INFO:teuthology.orchestra.run.smithi003.stdout: "user_id": "foobar.client.0", 2022-11-28T01:17:25.956 INFO:teuthology.orchestra.run.smithi003.stdout: "display_name": "Mr. foobar.client.0", 2022-11-28T01:17:25.956 INFO:teuthology.orchestra.run.smithi003.stdout: "email": "foobar.client.0+test@test.test", 2022-11-28T01:17:25.956 INFO:teuthology.orchestra.run.smithi003.stdout: "suspended": 0, 2022-11-28T01:17:25.957 INFO:teuthology.orchestra.run.smithi003.stdout: "max_buckets": 1000, 2022-11-28T01:17:25.957 INFO:teuthology.orchestra.run.smithi003.stdout: "subusers": [], 2022-11-28T01:17:25.957 INFO:teuthology.orchestra.run.smithi003.stdout: "keys": [ 2022-11-28T01:17:25.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:25.958 INFO:teuthology.orchestra.run.smithi003.stdout: "user": "foobar.client.0", 2022-11-28T01:17:25.958 INFO:teuthology.orchestra.run.smithi003.stdout: "access_key": "FQXPRDMECGUAMFUROOWN", 2022-11-28T01:17:25.958 INFO:teuthology.orchestra.run.smithi003.stdout: "secret_key": "3mO7d1HJbRSyuY3dzduv1pB/TPos68GWMP0ulZiW1LvKYsdF564MEA==" 2022-11-28T01:17:25.958 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:25.959 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:25.959 INFO:teuthology.orchestra.run.smithi003.stdout: "swift_keys": [], 2022-11-28T01:17:25.959 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": [ 2022-11-28T01:17:25.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:25.960 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "user-policy", 2022-11-28T01:17:25.960 INFO:teuthology.orchestra.run.smithi003.stdout: "perm": "*" 2022-11-28T01:17:25.960 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:25.960 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:25.961 INFO:teuthology.orchestra.run.smithi003.stdout: "op_mask": "read, write, delete", 2022-11-28T01:17:25.961 INFO:teuthology.orchestra.run.smithi003.stdout: "default_placement": "", 2022-11-28T01:17:25.961 INFO:teuthology.orchestra.run.smithi003.stdout: "default_storage_class": "", 2022-11-28T01:17:25.961 INFO:teuthology.orchestra.run.smithi003.stdout: "placement_tags": [], 2022-11-28T01:17:25.962 INFO:teuthology.orchestra.run.smithi003.stdout: "bucket_quota": { 2022-11-28T01:17:25.962 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": false, 2022-11-28T01:17:25.962 INFO:teuthology.orchestra.run.smithi003.stdout: "check_on_raw": false, 2022-11-28T01:17:25.962 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size": -1, 2022-11-28T01:17:25.963 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size_kb": 0, 2022-11-28T01:17:25.963 INFO:teuthology.orchestra.run.smithi003.stdout: "max_objects": -1 2022-11-28T01:17:25.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:25.963 INFO:teuthology.orchestra.run.smithi003.stdout: "user_quota": { 2022-11-28T01:17:25.964 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": false, 2022-11-28T01:17:25.964 INFO:teuthology.orchestra.run.smithi003.stdout: "check_on_raw": false, 2022-11-28T01:17:25.964 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size": -1, 2022-11-28T01:17:25.964 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size_kb": 0, 2022-11-28T01:17:25.965 INFO:teuthology.orchestra.run.smithi003.stdout: "max_objects": -1 2022-11-28T01:17:25.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:25.965 INFO:teuthology.orchestra.run.smithi003.stdout: "temp_url_keys": [], 2022-11-28T01:17:25.965 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "rgw", 2022-11-28T01:17:25.966 INFO:teuthology.orchestra.run.smithi003.stdout: "mfa_ids": [] 2022-11-28T01:17:25.966 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-11-28T01:17:25.966 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:25.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.947+0000 7faf38595700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:25.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.949+0000 7faf53619780 20 remove_watcher() i=0 2022-11-28T01:17:25.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.949+0000 7faf53619780 2 removed watcher, disabling cache 2022-11-28T01:17:25.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.949+0000 7faf53619780 20 remove_watcher() i=1 2022-11-28T01:17:25.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.950+0000 7faf53619780 20 remove_watcher() i=2 2022-11-28T01:17:25.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.951+0000 7faf53619780 20 remove_watcher() i=3 2022-11-28T01:17:25.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.952+0000 7faf53619780 20 remove_watcher() i=4 2022-11-28T01:17:25.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.953+0000 7faf53619780 20 remove_watcher() i=5 2022-11-28T01:17:25.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.954+0000 7faf53619780 20 remove_watcher() i=6 2022-11-28T01:17:25.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:25.954+0000 7faf53619780 20 remove_watcher() i=7 2022-11-28T01:17:25.972 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foobar.client.0 --totp-serial 8972148589 --totp-seed 4NOZ2QAIXADSWHMK46D7O3FTWFS6SSV5AO4KNZG5FFQFXZXGZHJAS3HGXY3FLYRS --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2022-11-28T01:17:26.027 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:26.027 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:26.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.034+0000 7fe4d4361780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.035+0000 7fe4d4361780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.035+0000 7fe4d4361780 20 realm 2022-11-28T01:17:26.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.035+0000 7fe4d4361780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.035+0000 7fe4d4361780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.035+0000 7fe4d4361780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:17:26.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.035+0000 7fe4d4361780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.036+0000 7fe4d4361780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.036+0000 7fe4d4361780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.036+0000 7fe4d4361780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:26.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.036+0000 7fe4d4361780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.036+0000 7fe4d4361780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:17:26.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.036+0000 7fe4d4361780 20 searching for the correct realm 2022-11-28T01:17:26.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.043+0000 7fe4d4361780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:17:26.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.043+0000 7fe4d4361780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:17:26.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.043+0000 7fe4d4361780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:17:26.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.043+0000 7fe4d4361780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:17:26.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.043+0000 7fe4d4361780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.043+0000 7fe4d4361780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.043+0000 7fe4d4361780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.043+0000 7fe4d4361780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:26.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.043+0000 7fe4d4361780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:17:26.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 20 zone default found 2022-11-28T01:17:26.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 4 Realm: () 2022-11-28T01:17:26.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:17:26.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:17:26.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:17:26.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 20 zonegroup default 2022-11-28T01:17:26.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 20 started sync module instance, tier type = 2022-11-28T01:17:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.044+0000 7fe4d4361780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:17:26.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.070+0000 7fe4d4361780 20 add_watcher() i=0 2022-11-28T01:17:26.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.070+0000 7fe4d4361780 20 add_watcher() i=1 2022-11-28T01:17:26.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.070+0000 7fe4d4361780 20 add_watcher() i=2 2022-11-28T01:17:26.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.070+0000 7fe4d4361780 20 add_watcher() i=3 2022-11-28T01:17:26.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.070+0000 7fe4d4361780 20 add_watcher() i=4 2022-11-28T01:17:26.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.070+0000 7fe4d4361780 20 add_watcher() i=5 2022-11-28T01:17:26.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.070+0000 7fe4d4361780 20 add_watcher() i=6 2022-11-28T01:17:26.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.073+0000 7fe4d4361780 20 add_watcher() i=7 2022-11-28T01:17:26.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.073+0000 7fe4d4361780 2 all 8 watchers are set, enabling cache 2022-11-28T01:17:26.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.076+0000 7fe4b92dd700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:26.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.076+0000 7fe4d4361780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:26.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.076+0000 7fe4d4361780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.080+0000 7fe4d4361780 5 note: GC not initialized 2022-11-28T01:17:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.081+0000 7fe4b4ad4700 20 reqs_thread_entry: start 2022-11-28T01:17:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.081+0000 7fe4d4361780 20 init_complete bucket index max shards: 11 2022-11-28T01:17:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.081+0000 7fe4d4361780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:17:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.081+0000 7fe4b2ad0700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:17:56 2022 2022-11-28T01:17:26.083 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:17:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.081+0000 7fe4b22ae700 20 reqs_thread_entry: start 2022-11-28T01:17:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.081+0000 7fe4d4361780 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2022-11-28T01:17:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.081+0000 7fe4d4361780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.082+0000 7fe4d4361780 20 rados_obj.operate() r=0 bl.length=467 2022-11-28T01:17:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.082+0000 7fe4d4361780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2022-11-28T01:17:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.082+0000 7fe4d4361780 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:17:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.082+0000 7fe4d4361780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2022-11-28T01:17:26.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.084+0000 7fe4d4361780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2022-11-28T01:17:26.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.084+0000 7fe4d4361780 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:17:26.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.084+0000 7fe4d4361780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2022-11-28T01:17:26.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.084+0000 7fe4ccb04700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117000 cookie 94376829811328 notifier 4308 bl.length()=669 2022-11-28T01:17:26.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.084+0000 7fe4ccb04700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2022-11-28T01:17:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.084+0000 7fe4ccb04700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:17:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.085+0000 7fe4b92dd700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.087+0000 7fe4d4361780 20 remove_watcher() i=0 2022-11-28T01:17:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.087+0000 7fe4d4361780 2 removed watcher, disabling cache 2022-11-28T01:17:26.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.088+0000 7fe4d4361780 20 remove_watcher() i=1 2022-11-28T01:17:26.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.089+0000 7fe4d4361780 20 remove_watcher() i=2 2022-11-28T01:17:26.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.090+0000 7fe4d4361780 20 remove_watcher() i=3 2022-11-28T01:17:26.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.092+0000 7fe4d4361780 20 remove_watcher() i=4 2022-11-28T01:17:26.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.093+0000 7fe4d4361780 20 remove_watcher() i=5 2022-11-28T01:17:26.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.095+0000 7fe4d4361780 20 remove_watcher() i=6 2022-11-28T01:17:26.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.096+0000 7fe4d4361780 20 remove_watcher() i=7 2022-11-28T01:17:26.101 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 caps add --uid foobar.client.0 --caps 'roles=*' --cluster ceph 2022-11-28T01:17:26.147 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:26.148 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:26.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.155+0000 7f16e1bc8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.156+0000 7f16e1bc8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.156+0000 7f16e1bc8780 20 realm 2022-11-28T01:17:26.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.156+0000 7f16e1bc8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.156+0000 7f16e1bc8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.156+0000 7f16e1bc8780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:17:26.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.156+0000 7f16e1bc8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.156+0000 7f16e1bc8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.156+0000 7f16e1bc8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.156+0000 7f16e1bc8780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:26.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.156+0000 7f16e1bc8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.157+0000 7f16e1bc8780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:17:26.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.157+0000 7f16e1bc8780 20 searching for the correct realm 2022-11-28T01:17:26.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.164+0000 7f16e1bc8780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:17:26.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.164+0000 7f16e1bc8780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:17:26.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.164+0000 7f16e1bc8780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:17:26.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.164+0000 7f16e1bc8780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:17:26.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.164+0000 7f16e1bc8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.164+0000 7f16e1bc8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.164+0000 7f16e1bc8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:26.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:17:26.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 zone default found 2022-11-28T01:17:26.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 4 Realm: () 2022-11-28T01:17:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:17:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:17:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:17:26.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 zonegroup default 2022-11-28T01:17:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 started sync module instance, tier type = 2022-11-28T01:17:26.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.165+0000 7f16e1bc8780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:17:26.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.193+0000 7f16e1bc8780 20 add_watcher() i=0 2022-11-28T01:17:26.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.193+0000 7f16e1bc8780 20 add_watcher() i=1 2022-11-28T01:17:26.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.193+0000 7f16e1bc8780 20 add_watcher() i=2 2022-11-28T01:17:26.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.193+0000 7f16e1bc8780 20 add_watcher() i=3 2022-11-28T01:17:26.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.193+0000 7f16e1bc8780 20 add_watcher() i=4 2022-11-28T01:17:26.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.193+0000 7f16e1bc8780 20 add_watcher() i=5 2022-11-28T01:17:26.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.193+0000 7f16e1bc8780 20 add_watcher() i=6 2022-11-28T01:17:26.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.195+0000 7f16e1bc8780 20 add_watcher() i=7 2022-11-28T01:17:26.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.196+0000 7f16e1bc8780 2 all 8 watchers are set, enabling cache 2022-11-28T01:17:26.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.198+0000 7f16c6b44700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:26.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.198+0000 7f16e1bc8780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:26.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.198+0000 7f16e1bc8780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:26.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.202+0000 7f16e1bc8780 5 note: GC not initialized 2022-11-28T01:17:26.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.203+0000 7f16c233b700 20 reqs_thread_entry: start 2022-11-28T01:17:26.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.203+0000 7f16e1bc8780 20 init_complete bucket index max shards: 11 2022-11-28T01:17:26.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.203+0000 7f16e1bc8780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:17:26.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.203+0000 7f16c0337700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:17:56 2022 2022-11-28T01:17:26.206 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:17:26.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.203+0000 7f16bfb15700 20 reqs_thread_entry: start 2022-11-28T01:17:26.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.203+0000 7f16e1bc8780 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2022-11-28T01:17:26.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.203+0000 7f16e1bc8780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.203+0000 7f16e1bc8780 20 rados_obj.operate() r=0 bl.length=481 2022-11-28T01:17:26.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.203+0000 7f16e1bc8780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2022-11-28T01:17:26.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.203+0000 7f16e1bc8780 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:17:26.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.203+0000 7f16e1bc8780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2022-11-28T01:17:26.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.205+0000 7f16e1bc8780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2022-11-28T01:17:26.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.205+0000 7f16e1bc8780 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:17:26.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.205+0000 7f16e1bc8780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2022-11-28T01:17:26.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.205+0000 7f16da36b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117001 cookie 94337208031872 notifier 4316 bl.length()=682 2022-11-28T01:17:26.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.205+0000 7f16da36b700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2022-11-28T01:17:26.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.205+0000 7f16da36b700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:17:26.210 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-11-28T01:17:26.211 INFO:teuthology.orchestra.run.smithi003.stdout: "user_id": "foobar.client.0", 2022-11-28T01:17:26.211 INFO:teuthology.orchestra.run.smithi003.stdout: "display_name": "Mr. foobar.client.0", 2022-11-28T01:17:26.211 INFO:teuthology.orchestra.run.smithi003.stdout: "email": "foobar.client.0+test@test.test", 2022-11-28T01:17:26.211 INFO:teuthology.orchestra.run.smithi003.stdout: "suspended": 0, 2022-11-28T01:17:26.212 INFO:teuthology.orchestra.run.smithi003.stdout: "max_buckets": 1000, 2022-11-28T01:17:26.212 INFO:teuthology.orchestra.run.smithi003.stdout: "subusers": [], 2022-11-28T01:17:26.212 INFO:teuthology.orchestra.run.smithi003.stdout: "keys": [ 2022-11-28T01:17:26.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:26.213 INFO:teuthology.orchestra.run.smithi003.stdout: "user": "foobar.client.0", 2022-11-28T01:17:26.213 INFO:teuthology.orchestra.run.smithi003.stdout: "access_key": "FQXPRDMECGUAMFUROOWN", 2022-11-28T01:17:26.213 INFO:teuthology.orchestra.run.smithi003.stdout: "secret_key": "3mO7d1HJbRSyuY3dzduv1pB/TPos68GWMP0ulZiW1LvKYsdF564MEA==" 2022-11-28T01:17:26.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:26.213 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:26.214 INFO:teuthology.orchestra.run.smithi003.stdout: "swift_keys": [], 2022-11-28T01:17:26.214 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": [ 2022-11-28T01:17:26.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:26.214 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "roles", 2022-11-28T01:17:26.215 INFO:teuthology.orchestra.run.smithi003.stdout: "perm": "*" 2022-11-28T01:17:26.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:26.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:26.215 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "user-policy", 2022-11-28T01:17:26.216 INFO:teuthology.orchestra.run.smithi003.stdout: "perm": "*" 2022-11-28T01:17:26.216 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:26.216 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:26.216 INFO:teuthology.orchestra.run.smithi003.stdout: "op_mask": "read, write, delete", 2022-11-28T01:17:26.216 INFO:teuthology.orchestra.run.smithi003.stdout: "default_placement": "", 2022-11-28T01:17:26.217 INFO:teuthology.orchestra.run.smithi003.stdout: "default_storage_class": "", 2022-11-28T01:17:26.217 INFO:teuthology.orchestra.run.smithi003.stdout: "placement_tags": [], 2022-11-28T01:17:26.217 INFO:teuthology.orchestra.run.smithi003.stdout: "bucket_quota": { 2022-11-28T01:17:26.217 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": false, 2022-11-28T01:17:26.218 INFO:teuthology.orchestra.run.smithi003.stdout: "check_on_raw": false, 2022-11-28T01:17:26.218 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size": -1, 2022-11-28T01:17:26.218 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size_kb": 0, 2022-11-28T01:17:26.218 INFO:teuthology.orchestra.run.smithi003.stdout: "max_objects": -1 2022-11-28T01:17:26.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:26.219 INFO:teuthology.orchestra.run.smithi003.stdout: "user_quota": { 2022-11-28T01:17:26.219 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": false, 2022-11-28T01:17:26.219 INFO:teuthology.orchestra.run.smithi003.stdout: "check_on_raw": false, 2022-11-28T01:17:26.220 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size": -1, 2022-11-28T01:17:26.220 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size_kb": 0, 2022-11-28T01:17:26.220 INFO:teuthology.orchestra.run.smithi003.stdout: "max_objects": -1 2022-11-28T01:17:26.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:26.220 INFO:teuthology.orchestra.run.smithi003.stdout: "temp_url_keys": [], 2022-11-28T01:17:26.221 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "rgw", 2022-11-28T01:17:26.221 INFO:teuthology.orchestra.run.smithi003.stdout: "mfa_ids": [ 2022-11-28T01:17:26.221 INFO:teuthology.orchestra.run.smithi003.stdout: "8972148589" 2022-11-28T01:17:26.221 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-11-28T01:17:26.222 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-11-28T01:17:26.222 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:26.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.206+0000 7f16c6b44700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:26.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.210+0000 7f16e1bc8780 20 remove_watcher() i=0 2022-11-28T01:17:26.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.210+0000 7f16e1bc8780 2 removed watcher, disabling cache 2022-11-28T01:17:26.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.213+0000 7f16e1bc8780 20 remove_watcher() i=1 2022-11-28T01:17:26.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.215+0000 7f16e1bc8780 20 remove_watcher() i=2 2022-11-28T01:17:26.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.218+0000 7f16e1bc8780 20 remove_watcher() i=3 2022-11-28T01:17:26.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.220+0000 7f16e1bc8780 20 remove_watcher() i=4 2022-11-28T01:17:26.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.221+0000 7f16e1bc8780 20 remove_watcher() i=5 2022-11-28T01:17:26.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.223+0000 7f16e1bc8780 20 remove_watcher() i=6 2022-11-28T01:17:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.225+0000 7f16e1bc8780 20 remove_watcher() i=7 2022-11-28T01:17:26.231 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 caps add --uid foobar.client.0 --caps 'oidc-provider=*' --cluster ceph 2022-11-28T01:17:26.278 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:17:26.278 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:17:26.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.293+0000 7fb71ec31780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.297+0000 7fb71ec31780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.297+0000 7fb71ec31780 20 realm 2022-11-28T01:17:26.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.297+0000 7fb71ec31780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.298+0000 7fb71ec31780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.298+0000 7fb71ec31780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:17:26.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.298+0000 7fb71ec31780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.298+0000 7fb71ec31780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.298+0000 7fb71ec31780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.300+0000 7fb71ec31780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:26.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.300+0000 7fb71ec31780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.300+0000 7fb71ec31780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:17:26.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.301+0000 7fb71ec31780 20 searching for the correct realm 2022-11-28T01:17:26.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.312+0000 7fb71ec31780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:17:26.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.312+0000 7fb71ec31780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:17:26.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.312+0000 7fb71ec31780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:17:26.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.312+0000 7fb71ec31780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:17:26.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.312+0000 7fb71ec31780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.312+0000 7fb71ec31780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.312+0000 7fb71ec31780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:17:26.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:17:26.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 20 zone default found 2022-11-28T01:17:26.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 4 Realm: () 2022-11-28T01:17:26.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:17:26.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:17:26.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:17:26.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 20 zonegroup default 2022-11-28T01:17:26.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.313+0000 7fb71ec31780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.314+0000 7fb71ec31780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:17:26.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.314+0000 7fb71ec31780 20 started sync module instance, tier type = 2022-11-28T01:17:26.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.314+0000 7fb71ec31780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:17:26.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.329+0000 7fb71ec31780 20 add_watcher() i=0 2022-11-28T01:17:26.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.329+0000 7fb71ec31780 20 add_watcher() i=1 2022-11-28T01:17:26.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.329+0000 7fb71ec31780 20 add_watcher() i=2 2022-11-28T01:17:26.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.329+0000 7fb71ec31780 20 add_watcher() i=3 2022-11-28T01:17:26.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.329+0000 7fb71ec31780 20 add_watcher() i=4 2022-11-28T01:17:26.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.329+0000 7fb71ec31780 20 add_watcher() i=5 2022-11-28T01:17:26.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.329+0000 7fb71ec31780 20 add_watcher() i=6 2022-11-28T01:17:26.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.330+0000 7fb71ec31780 20 add_watcher() i=7 2022-11-28T01:17:26.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.330+0000 7fb71ec31780 2 all 8 watchers are set, enabling cache 2022-11-28T01:17:26.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.332+0000 7fb703bad700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:26.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.332+0000 7fb71ec31780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:17:26.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.332+0000 7fb71ec31780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:17:26.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.335+0000 7fb71ec31780 5 note: GC not initialized 2022-11-28T01:17:26.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.336+0000 7fb6ff3a4700 20 reqs_thread_entry: start 2022-11-28T01:17:26.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.336+0000 7fb71ec31780 20 init_complete bucket index max shards: 11 2022-11-28T01:17:26.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.337+0000 7fb71ec31780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:17:26.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.337+0000 7fb6fd3a0700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:17:56 2022 2022-11-28T01:17:26.339 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:17:26.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.337+0000 7fb6fcb7e700 20 reqs_thread_entry: start 2022-11-28T01:17:26.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.337+0000 7fb71ec31780 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2022-11-28T01:17:26.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.337+0000 7fb71ec31780 20 rados->read ofs=0 len=0 2022-11-28T01:17:26.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.337+0000 7fb71ec31780 20 rados_obj.operate() r=0 bl.length=494 2022-11-28T01:17:26.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.337+0000 7fb71ec31780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2022-11-28T01:17:26.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.337+0000 7fb71ec31780 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:17:26.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.337+0000 7fb71ec31780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2022-11-28T01:17:26.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.339+0000 7fb71ec31780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2022-11-28T01:17:26.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.339+0000 7fb71ec31780 10 moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:17:26.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.339+0000 7fb71ec31780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2022-11-28T01:17:26.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.339+0000 7fb7173d4700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084298 cookie 93992277064320 notifier 4324 bl.length()=703 2022-11-28T01:17:26.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.339+0000 7fb7173d4700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x17 2022-11-28T01:17:26.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.339+0000 7fb7173d4700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:17:26.343 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-11-28T01:17:26.344 INFO:teuthology.orchestra.run.smithi003.stdout: "user_id": "foobar.client.0", 2022-11-28T01:17:26.344 INFO:teuthology.orchestra.run.smithi003.stdout: "display_name": "Mr. foobar.client.0", 2022-11-28T01:17:26.344 INFO:teuthology.orchestra.run.smithi003.stdout: "email": "foobar.client.0+test@test.test", 2022-11-28T01:17:26.344 INFO:teuthology.orchestra.run.smithi003.stdout: "suspended": 0, 2022-11-28T01:17:26.344 INFO:teuthology.orchestra.run.smithi003.stdout: "max_buckets": 1000, 2022-11-28T01:17:26.345 INFO:teuthology.orchestra.run.smithi003.stdout: "subusers": [], 2022-11-28T01:17:26.345 INFO:teuthology.orchestra.run.smithi003.stdout: "keys": [ 2022-11-28T01:17:26.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:26.345 INFO:teuthology.orchestra.run.smithi003.stdout: "user": "foobar.client.0", 2022-11-28T01:17:26.346 INFO:teuthology.orchestra.run.smithi003.stdout: "access_key": "FQXPRDMECGUAMFUROOWN", 2022-11-28T01:17:26.346 INFO:teuthology.orchestra.run.smithi003.stdout: "secret_key": "3mO7d1HJbRSyuY3dzduv1pB/TPos68GWMP0ulZiW1LvKYsdF564MEA==" 2022-11-28T01:17:26.346 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:26.346 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:26.346 INFO:teuthology.orchestra.run.smithi003.stdout: "swift_keys": [], 2022-11-28T01:17:26.347 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": [ 2022-11-28T01:17:26.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:26.347 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "oidc-provider", 2022-11-28T01:17:26.347 INFO:teuthology.orchestra.run.smithi003.stdout: "perm": "*" 2022-11-28T01:17:26.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:26.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:26.348 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "roles", 2022-11-28T01:17:26.348 INFO:teuthology.orchestra.run.smithi003.stdout: "perm": "*" 2022-11-28T01:17:26.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:26.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-11-28T01:17:26.349 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "user-policy", 2022-11-28T01:17:26.349 INFO:teuthology.orchestra.run.smithi003.stdout: "perm": "*" 2022-11-28T01:17:26.349 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-11-28T01:17:26.349 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-11-28T01:17:26.350 INFO:teuthology.orchestra.run.smithi003.stdout: "op_mask": "read, write, delete", 2022-11-28T01:17:26.350 INFO:teuthology.orchestra.run.smithi003.stdout: "default_placement": "", 2022-11-28T01:17:26.350 INFO:teuthology.orchestra.run.smithi003.stdout: "default_storage_class": "", 2022-11-28T01:17:26.350 INFO:teuthology.orchestra.run.smithi003.stdout: "placement_tags": [], 2022-11-28T01:17:26.350 INFO:teuthology.orchestra.run.smithi003.stdout: "bucket_quota": { 2022-11-28T01:17:26.351 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": false, 2022-11-28T01:17:26.351 INFO:teuthology.orchestra.run.smithi003.stdout: "check_on_raw": false, 2022-11-28T01:17:26.351 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size": -1, 2022-11-28T01:17:26.351 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size_kb": 0, 2022-11-28T01:17:26.352 INFO:teuthology.orchestra.run.smithi003.stdout: "max_objects": -1 2022-11-28T01:17:26.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:26.352 INFO:teuthology.orchestra.run.smithi003.stdout: "user_quota": { 2022-11-28T01:17:26.352 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": false, 2022-11-28T01:17:26.352 INFO:teuthology.orchestra.run.smithi003.stdout: "check_on_raw": false, 2022-11-28T01:17:26.353 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size": -1, 2022-11-28T01:17:26.353 INFO:teuthology.orchestra.run.smithi003.stdout: "max_size_kb": 0, 2022-11-28T01:17:26.353 INFO:teuthology.orchestra.run.smithi003.stdout: "max_objects": -1 2022-11-28T01:17:26.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-11-28T01:17:26.354 INFO:teuthology.orchestra.run.smithi003.stdout: "temp_url_keys": [], 2022-11-28T01:17:26.354 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "rgw", 2022-11-28T01:17:26.354 INFO:teuthology.orchestra.run.smithi003.stdout: "mfa_ids": [ 2022-11-28T01:17:26.354 INFO:teuthology.orchestra.run.smithi003.stdout: "8972148589" 2022-11-28T01:17:26.354 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-11-28T01:17:26.355 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-11-28T01:17:26.355 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:26.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.340+0000 7fb703bad700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:17:26.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.342+0000 7fb71ec31780 20 remove_watcher() i=0 2022-11-28T01:17:26.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.342+0000 7fb71ec31780 2 removed watcher, disabling cache 2022-11-28T01:17:26.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.343+0000 7fb71ec31780 20 remove_watcher() i=1 2022-11-28T01:17:26.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.343+0000 7fb71ec31780 20 remove_watcher() i=2 2022-11-28T01:17:26.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.344+0000 7fb71ec31780 20 remove_watcher() i=3 2022-11-28T01:17:26.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.345+0000 7fb71ec31780 20 remove_watcher() i=4 2022-11-28T01:17:26.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.346+0000 7fb71ec31780 20 remove_watcher() i=5 2022-11-28T01:17:26.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.347+0000 7fb71ec31780 20 remove_watcher() i=6 2022-11-28T01:17:26.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:17:26.348+0000 7fb71ec31780 20 remove_watcher() i=7 2022-11-28T01:17:26.359 INFO:tasks.s3tests:Configuring s3-tests... 2022-11-28T01:17:26.360 DEBUG:teuthology.orchestra.run.smithi003:> cd /home/ubuntu/cephtest/s3-tests-client.0 && ./bootstrap 2022-11-28T01:17:33.964 INFO:teuthology.orchestra.run.smithi003.stderr:./bootstrap: Missing required RPM packages: libevent-devel libffi-devel libxml2-devel libxslt-devel zlib-devel. 2022-11-28T01:17:34.580 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:03:34 ago on Mon 28 Nov 2022 01:14:00 AM UTC. 2022-11-28T01:17:35.748 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2022-11-28T01:17:35.749 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:17:35.750 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2022-11-28T01:17:35.750 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:17:35.750 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2022-11-28T01:17:35.750 INFO:teuthology.orchestra.run.smithi003.stdout: libevent-devel x86_64 2.1.8-5.el8 CentOS-AppStream 104 k 2022-11-28T01:17:35.751 INFO:teuthology.orchestra.run.smithi003.stdout: libffi-devel x86_64 3.1-23.el8 CentOS-BaseOS 29 k 2022-11-28T01:17:35.751 INFO:teuthology.orchestra.run.smithi003.stdout: libxml2-devel x86_64 2.9.7-15.el8 CentOS-AppStream 1.0 M 2022-11-28T01:17:35.751 INFO:teuthology.orchestra.run.smithi003.stdout: libxslt-devel x86_64 1.1.32-6.el8 CentOS-AppStream 323 k 2022-11-28T01:17:35.752 INFO:teuthology.orchestra.run.smithi003.stdout: zlib-devel x86_64 1.2.11-20.el8 CentOS-BaseOS 58 k 2022-11-28T01:17:35.752 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2022-11-28T01:17:35.753 INFO:teuthology.orchestra.run.smithi003.stdout: cmake-filesystem x86_64 3.20.2-4.el8 CentOS-AppStream 44 k 2022-11-28T01:17:35.753 INFO:teuthology.orchestra.run.smithi003.stdout: libgcrypt-devel x86_64 1.8.5-7.el8 CentOS-BaseOS 150 k 2022-11-28T01:17:35.753 INFO:teuthology.orchestra.run.smithi003.stdout: libgpg-error-devel x86_64 1.31-1.el8 CentOS-BaseOS 64 k 2022-11-28T01:17:35.754 INFO:teuthology.orchestra.run.smithi003.stdout: xz-devel x86_64 5.2.4-4.el8 CentOS-BaseOS 63 k 2022-11-28T01:17:35.754 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:35.755 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2022-11-28T01:17:35.755 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2022-11-28T01:17:35.755 INFO:teuthology.orchestra.run.smithi003.stdout:Install 9 Packages 2022-11-28T01:17:35.755 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:35.756 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 1.9 M 2022-11-28T01:17:35.757 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 12 M 2022-11-28T01:17:35.757 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2022-11-28T01:17:35.815 INFO:teuthology.orchestra.run.smithi003.stdout:(1/9): cmake-filesystem-3.20.2-4.el8.x86_64.rpm 718 kB/s | 44 kB 00:00 2022-11-28T01:17:35.848 INFO:teuthology.orchestra.run.smithi003.stdout:(2/9): libevent-devel-2.1.8-5.el8.x86_64.rpm 1.1 MB/s | 104 kB 00:00 2022-11-28T01:17:35.865 INFO:teuthology.orchestra.run.smithi003.stdout:(3/9): libxslt-devel-1.1.32-6.el8.x86_64.rpm 6.4 MB/s | 323 kB 00:00 2022-11-28T01:17:35.882 INFO:teuthology.orchestra.run.smithi003.stdout:(4/9): libffi-devel-3.1-23.el8.x86_64.rpm 856 kB/s | 29 kB 00:00 2022-11-28T01:17:35.916 INFO:teuthology.orchestra.run.smithi003.stdout:(5/9): libxml2-devel-2.9.7-15.el8.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2022-11-28T01:17:35.941 INFO:teuthology.orchestra.run.smithi003.stdout:(6/9): libgcrypt-devel-1.8.5-7.el8.x86_64.rpm 1.9 MB/s | 150 kB 00:00 2022-11-28T01:17:35.966 INFO:teuthology.orchestra.run.smithi003.stdout:(7/9): libgpg-error-devel-1.31-1.el8.x86_64.rpm 761 kB/s | 64 kB 00:00 2022-11-28T01:17:35.992 INFO:teuthology.orchestra.run.smithi003.stdout:(8/9): xz-devel-5.2.4-4.el8.x86_64.rpm 833 kB/s | 63 kB 00:00 2022-11-28T01:17:36.008 INFO:teuthology.orchestra.run.smithi003.stdout:(9/9): zlib-devel-1.2.11-20.el8.x86_64.rpm 870 kB/s | 58 kB 00:00 2022-11-28T01:17:36.010 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2022-11-28T01:17:36.011 INFO:teuthology.orchestra.run.smithi003.stdout:Total 7.1 MB/s | 1.9 MB 00:00 2022-11-28T01:17:36.093 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2022-11-28T01:17:36.105 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2022-11-28T01:17:36.106 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2022-11-28T01:17:36.246 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2022-11-28T01:17:36.249 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2022-11-28T01:17:36.608 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2022-11-28T01:17:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 1/9 2022-11-28T01:17:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 1/9 2022-11-28T01:17:36.907 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgcrypt-devel-1.8.5-7.el8.x86_64 2/9 2022-11-28T01:17:36.919 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libgcrypt-devel-1.8.5-7.el8.x86_64 2/9 2022-11-28T01:17:37.130 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : zlib-devel-1.2.11-20.el8.x86_64 3/9 2022-11-28T01:17:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xz-devel-5.2.4-4.el8.x86_64 4/9 2022-11-28T01:17:37.768 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cmake-filesystem-3.20.2-4.el8.x86_64 5/9 2022-11-28T01:17:38.103 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libxml2-devel-2.9.7-15.el8.x86_64 6/9 2022-11-28T01:17:38.346 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 7/9 2022-11-28T01:17:38.582 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libffi-devel-3.1-23.el8.x86_64 8/9 2022-11-28T01:17:38.612 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libffi-devel-3.1-23.el8.x86_64 8/9 2022-11-28T01:17:38.806 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 9/9 2022-11-28T01:17:39.085 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libevent-devel-2.1.8-5.el8.x86_64 9/9 2022-11-28T01:17:39.085 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cmake-filesystem-3.20.2-4.el8.x86_64 1/9 2022-11-28T01:17:39.085 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 2/9 2022-11-28T01:17:39.086 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libxml2-devel-2.9.7-15.el8.x86_64 3/9 2022-11-28T01:17:39.086 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 4/9 2022-11-28T01:17:39.087 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libffi-devel-3.1-23.el8.x86_64 5/9 2022-11-28T01:17:39.087 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgcrypt-devel-1.8.5-7.el8.x86_64 6/9 2022-11-28T01:17:39.087 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 7/9 2022-11-28T01:17:39.088 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xz-devel-5.2.4-4.el8.x86_64 8/9 2022-11-28T01:17:39.593 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : zlib-devel-1.2.11-20.el8.x86_64 9/9 2022-11-28T01:17:39.593 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:39.593 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2022-11-28T01:17:39.593 INFO:teuthology.orchestra.run.smithi003.stdout: cmake-filesystem-3.20.2-4.el8.x86_64 libevent-devel-2.1.8-5.el8.x86_64 2022-11-28T01:17:39.594 INFO:teuthology.orchestra.run.smithi003.stdout: libffi-devel-3.1-23.el8.x86_64 libgcrypt-devel-1.8.5-7.el8.x86_64 2022-11-28T01:17:39.594 INFO:teuthology.orchestra.run.smithi003.stdout: libgpg-error-devel-1.31-1.el8.x86_64 libxml2-devel-2.9.7-15.el8.x86_64 2022-11-28T01:17:39.594 INFO:teuthology.orchestra.run.smithi003.stdout: libxslt-devel-1.1.32-6.el8.x86_64 xz-devel-5.2.4-4.el8.x86_64 2022-11-28T01:17:39.594 INFO:teuthology.orchestra.run.smithi003.stdout: zlib-devel-1.2.11-20.el8.x86_64 2022-11-28T01:17:39.595 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:39.595 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2022-11-28T01:17:40.126 INFO:teuthology.orchestra.run.smithi003.stdout:Already using interpreter /usr/bin/python3.6 2022-11-28T01:17:40.127 INFO:teuthology.orchestra.run.smithi003.stdout:Using base prefix '/usr' 2022-11-28T01:17:40.127 INFO:teuthology.orchestra.run.smithi003.stdout:New python executable in /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python3.6 2022-11-28T01:17:40.127 INFO:teuthology.orchestra.run.smithi003.stdout:Also creating executable in /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python 2022-11-28T01:17:43.654 INFO:teuthology.orchestra.run.smithi003.stdout:Installing setuptools, pip, wheel...done. 2022-11-28T01:17:44.010 INFO:teuthology.orchestra.run.smithi003.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.6/site-packages (21.3.1) 2022-11-28T01:17:45.374 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting setuptools==44.1.0 2022-11-28T01:17:45.425 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2022-11-28T01:17:45.527 INFO:teuthology.orchestra.run.smithi003.stdout:Installing collected packages: setuptools 2022-11-28T01:17:45.527 INFO:teuthology.orchestra.run.smithi003.stdout: Attempting uninstall: setuptools 2022-11-28T01:17:45.527 INFO:teuthology.orchestra.run.smithi003.stdout: Found existing installation: setuptools 59.6.0 2022-11-28T01:17:45.577 INFO:teuthology.orchestra.run.smithi003.stdout: Uninstalling setuptools-59.6.0: 2022-11-28T01:17:45.585 INFO:teuthology.orchestra.run.smithi003.stdout: Successfully uninstalled setuptools-59.6.0 2022-11-28T01:17:45.828 INFO:teuthology.orchestra.run.smithi003.stdout:Successfully installed setuptools-44.1.0 2022-11-28T01:17:46.430 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting PyYAML 2022-11-28T01:17:46.482 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (603 kB) 2022-11-28T01:17:46.601 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting nose>=1.0.0 2022-11-28T01:17:46.636 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2022-11-28T01:17:46.748 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting boto>=2.6.0 2022-11-28T01:17:46.777 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2022-11-28T01:17:47.663 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting boto3>=1.0.0 2022-11-28T01:17:47.684 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2022-11-28T01:17:48.695 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting botocore<1.28.0 2022-11-28T01:17:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2022-11-28T01:17:48.970 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting munch>=2.0.0 2022-11-28T01:17:48.981 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading munch-2.5.0-py2.py3-none-any.whl (10 kB) 2022-11-28T01:17:49.389 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting gevent>=1.0 2022-11-28T01:17:49.402 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading gevent-22.10.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB) 2022-11-28T01:17:49.580 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting isodate>=0.4.4 2022-11-28T01:17:49.592 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2022-11-28T01:17:49.713 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting requests>=2.23.0 2022-11-28T01:17:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2022-11-28T01:17:49.949 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting pytz>=2011k 2022-11-28T01:17:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading pytz-2022.6-py2.py3-none-any.whl (498 kB) 2022-11-28T01:17:50.071 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting httplib2 2022-11-28T01:17:50.083 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading httplib2-0.21.0-py3-none-any.whl (96 kB) 2022-11-28T01:17:50.654 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting lxml 2022-11-28T01:17:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading lxml-4.9.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.4 MB) 2022-11-28T01:17:51.044 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting jmespath<2.0.0,>=0.7.1 2022-11-28T01:17:51.057 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2022-11-28T01:17:51.134 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2022-11-28T01:17:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2022-11-28T01:17:51.285 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting urllib3<1.27,>=1.25.4 2022-11-28T01:17:51.299 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading urllib3-1.26.13-py2.py3-none-any.whl (140 kB) 2022-11-28T01:17:51.387 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting python-dateutil<3.0.0,>=2.1 2022-11-28T01:17:51.399 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2022-11-28T01:17:51.499 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting six 2022-11-28T01:17:51.513 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2022-11-28T01:17:51.991 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting zope.interface 2022-11-28T01:17:52.008 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253 kB) 2022-11-28T01:17:52.418 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting zope.event 2022-11-28T01:17:52.431 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading zope.event-4.5.0-py2.py3-none-any.whl (6.8 kB) 2022-11-28T01:17:52.590 INFO:teuthology.orchestra.run.smithi003.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (from gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2022-11-28T01:17:52.979 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting greenlet>=2.0.0 2022-11-28T01:17:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading greenlet-2.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (513 kB) 2022-11-28T01:17:53.139 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting certifi>=2017.4.17 2022-11-28T01:17:53.151 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading certifi-2022.9.24-py3-none-any.whl (161 kB) 2022-11-28T01:17:53.342 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting charset-normalizer~=2.0.0 2022-11-28T01:17:53.354 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2022-11-28T01:17:53.445 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting idna<4,>=2.5 2022-11-28T01:17:53.457 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading idna-3.4-py3-none-any.whl (61 kB) 2022-11-28T01:17:53.596 INFO:teuthology.orchestra.run.smithi003.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2022-11-28T01:17:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB) 2022-11-28T01:17:54.215 INFO:teuthology.orchestra.run.smithi003.stdout:Installing collected packages: six, urllib3, python-dateutil, jmespath, botocore, zope.interface, zope.event, s3transfer, pyparsing, idna, greenlet, charset-normalizer, certifi, requests, PyYAML, pytz, nose, munch, lxml, isodate, httplib2, gevent, boto3, boto 2022-11-28T01:17:57.876 INFO:teuthology.orchestra.run.smithi003.stdout:Successfully installed PyYAML-6.0 boto-2.49.0 boto3-1.23.10 botocore-1.26.10 certifi-2022.9.24 charset-normalizer-2.0.12 gevent-22.10.2 greenlet-2.0.1 httplib2-0.21.0 idna-3.4 isodate-0.6.1 jmespath-0.10.0 lxml-4.9.1 munch-2.5.0 nose-1.3.7 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2022.6 requests-2.27.1 s3transfer-0.5.2 six-1.16.0 urllib3-1.26.13 zope.event-4.5.0 zope.interface-5.5.2 2022-11-28T01:17:58.182 INFO:teuthology.orchestra.run.smithi003.stdout:running develop 2022-11-28T01:17:58.203 INFO:teuthology.orchestra.run.smithi003.stdout:running egg_info 2022-11-28T01:17:58.203 INFO:teuthology.orchestra.run.smithi003.stdout:creating s3tests.egg-info 2022-11-28T01:17:58.203 INFO:teuthology.orchestra.run.smithi003.stdout:writing s3tests.egg-info/PKG-INFO 2022-11-28T01:17:58.203 INFO:teuthology.orchestra.run.smithi003.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2022-11-28T01:17:58.204 INFO:teuthology.orchestra.run.smithi003.stdout:writing requirements to s3tests.egg-info/requires.txt 2022-11-28T01:17:58.204 INFO:teuthology.orchestra.run.smithi003.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2022-11-28T01:17:58.204 INFO:teuthology.orchestra.run.smithi003.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2022-11-28T01:17:58.214 INFO:teuthology.orchestra.run.smithi003.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2022-11-28T01:17:58.261 INFO:teuthology.orchestra.run.smithi003.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2022-11-28T01:17:58.261 INFO:teuthology.orchestra.run.smithi003.stdout:running build_ext 2022-11-28T01:17:58.261 INFO:teuthology.orchestra.run.smithi003.stdout:Creating /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages/s3tests.egg-link (link to .) 2022-11-28T01:17:58.262 INFO:teuthology.orchestra.run.smithi003.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2022-11-28T01:17:58.262 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.262 INFO:teuthology.orchestra.run.smithi003.stdout:Installed /home/ubuntu/cephtest/s3-tests-client.0 2022-11-28T01:17:58.263 INFO:teuthology.orchestra.run.smithi003.stdout:Processing dependencies for s3tests==0.0.1 2022-11-28T01:17:58.323 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for isodate==0.6.1 2022-11-28T01:17:58.324 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: isodate 0.6.1 2022-11-28T01:17:58.324 INFO:teuthology.orchestra.run.smithi003.stdout:Adding isodate 0.6.1 to easy-install.pth file 2022-11-28T01:17:58.324 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.325 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.325 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for gevent==22.10.2 2022-11-28T01:17:58.325 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: gevent 22.10.2 2022-11-28T01:17:58.325 INFO:teuthology.orchestra.run.smithi003.stdout:Adding gevent 22.10.2 to easy-install.pth file 2022-11-28T01:17:58.326 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.326 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.327 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for munch==2.5.0 2022-11-28T01:17:58.328 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: munch 2.5.0 2022-11-28T01:17:58.328 INFO:teuthology.orchestra.run.smithi003.stdout:Adding munch 2.5.0 to easy-install.pth file 2022-11-28T01:17:58.328 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.328 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.329 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for PyYAML==6.0 2022-11-28T01:17:58.329 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: PyYAML 6.0 2022-11-28T01:17:58.329 INFO:teuthology.orchestra.run.smithi003.stdout:Adding PyYAML 6.0 to easy-install.pth file 2022-11-28T01:17:58.330 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.330 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.330 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for boto3==1.23.10 2022-11-28T01:17:58.330 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: boto3 1.23.10 2022-11-28T01:17:58.336 INFO:teuthology.orchestra.run.smithi003.stdout:Adding boto3 1.23.10 to easy-install.pth file 2022-11-28T01:17:58.336 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.336 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.336 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for boto==2.49.0 2022-11-28T01:17:58.337 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: boto 2.49.0 2022-11-28T01:17:58.337 INFO:teuthology.orchestra.run.smithi003.stdout:Adding boto 2.49.0 to easy-install.pth file 2022-11-28T01:17:58.337 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.337 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.338 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for six==1.16.0 2022-11-28T01:17:58.338 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: six 1.16.0 2022-11-28T01:17:58.338 INFO:teuthology.orchestra.run.smithi003.stdout:Adding six 1.16.0 to easy-install.pth file 2022-11-28T01:17:58.338 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.338 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.339 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for zope.event==4.5.0 2022-11-28T01:17:58.339 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: zope.event 4.5.0 2022-11-28T01:17:58.339 INFO:teuthology.orchestra.run.smithi003.stdout:Adding zope.event 4.5.0 to easy-install.pth file 2022-11-28T01:17:58.339 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.340 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.340 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for greenlet==2.0.1 2022-11-28T01:17:58.340 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: greenlet 2.0.1 2022-11-28T01:17:58.341 INFO:teuthology.orchestra.run.smithi003.stdout:Adding greenlet 2.0.1 to easy-install.pth file 2022-11-28T01:17:58.341 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.341 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.341 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for zope.interface==5.5.2 2022-11-28T01:17:58.342 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: zope.interface 5.5.2 2022-11-28T01:17:58.342 INFO:teuthology.orchestra.run.smithi003.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2022-11-28T01:17:58.342 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.342 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.343 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for setuptools==44.1.0 2022-11-28T01:17:58.343 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: setuptools 44.1.0 2022-11-28T01:17:58.343 INFO:teuthology.orchestra.run.smithi003.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2022-11-28T01:17:58.343 INFO:teuthology.orchestra.run.smithi003.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin 2022-11-28T01:17:58.343 INFO:teuthology.orchestra.run.smithi003.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin 2022-11-28T01:17:58.344 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.344 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.344 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for jmespath==0.10.0 2022-11-28T01:17:58.344 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: jmespath 0.10.0 2022-11-28T01:17:58.344 INFO:teuthology.orchestra.run.smithi003.stdout:Adding jmespath 0.10.0 to easy-install.pth file 2022-11-28T01:17:58.345 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.345 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.345 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for botocore==1.26.10 2022-11-28T01:17:58.345 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: botocore 1.26.10 2022-11-28T01:17:58.346 INFO:teuthology.orchestra.run.smithi003.stdout:Adding botocore 1.26.10 to easy-install.pth file 2022-11-28T01:17:58.346 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.346 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.346 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for s3transfer==0.5.2 2022-11-28T01:17:58.346 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: s3transfer 0.5.2 2022-11-28T01:17:58.347 INFO:teuthology.orchestra.run.smithi003.stdout:Adding s3transfer 0.5.2 to easy-install.pth file 2022-11-28T01:17:58.347 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.347 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.348 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for urllib3==1.26.13 2022-11-28T01:17:58.348 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: urllib3 1.26.13 2022-11-28T01:17:58.348 INFO:teuthology.orchestra.run.smithi003.stdout:Adding urllib3 1.26.13 to easy-install.pth file 2022-11-28T01:17:58.348 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.349 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.350 INFO:teuthology.orchestra.run.smithi003.stdout:Searching for python-dateutil==2.8.2 2022-11-28T01:17:58.350 INFO:teuthology.orchestra.run.smithi003.stdout:Best match: python-dateutil 2.8.2 2022-11-28T01:17:58.350 INFO:teuthology.orchestra.run.smithi003.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2022-11-28T01:17:58.351 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:17:58.351 INFO:teuthology.orchestra.run.smithi003.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2022-11-28T01:17:58.351 INFO:teuthology.orchestra.run.smithi003.stdout:Finished processing dependencies for s3tests==0.0.1 2022-11-28T01:17:58.356 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:17:58.357 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2022-11-28T01:17:58.376 INFO:tasks.s3tests:Configuring boto... 2022-11-28T01:17:58.377 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-11-28T01:17:58.377 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/boto-client.0.cfg 2022-11-28T01:17:58.435 DEBUG:teuthology.orchestra.run.smithi003:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a lifecycle 2022-11-28T01:17:59.155 INFO:teuthology.orchestra.run.smithi003.stderr:/home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2022-11-28T01:17:59.155 INFO:teuthology.orchestra.run.smithi003.stderr: warnings.warn(warning, PythonDeprecationWarning) 2022-11-28T01:17:59.184 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2022-11-28T01:17:59.255 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2022-11-28T01:17:59.327 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2022-11-28T01:19:09.529 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration ... ok 2022-11-28T01:20:19.714 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecyclev2_expiration ... ok 2022-11-28T01:20:49.869 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_versioning_enabled ... ok 2022-11-28T01:21:19.962 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_tags1 ... ok 2022-11-28T01:21:50.086 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_tags2 ... ok 2022-11-28T01:22:20.254 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_versioned_tags2 ... ok 2022-11-28T01:23:30.626 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_noncur_tags1 ... ok 2022-11-28T01:23:30.657 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2022-11-28T01:23:30.687 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2022-11-28T01:23:30.803 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2022-11-28T01:23:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2022-11-28T01:23:30.862 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2022-11-28T01:24:00.979 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_date ... ok 2022-11-28T01:24:01.032 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_days0 ... ok 2022-11-28T01:24:01.079 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_header_put ... ok 2022-11-28T01:24:01.131 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_header_head ... ok 2022-11-28T01:24:01.204 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_header_tags_head ... ok 2022-11-28T01:24:01.261 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_expiration_header_and_tags_head ... ok 2022-11-28T01:24:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2022-11-28T01:24:51.564 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_noncur_expiration ... ok 2022-11-28T01:24:51.600 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2022-11-28T01:24:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2022-11-28T01:24:51.681 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2022-11-28T01:26:01.930 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_deletemarker_expiration ... ok 2022-11-28T01:26:01.970 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2022-11-28T01:26:53.999 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_multipart_expiration ... ok 2022-11-28T01:26:54.035 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_set_invalid_date ... ok 2022-11-28T01:28:34.284 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition ... ok 2022-11-28T01:30:34.551 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... ok 2022-11-28T01:30:34.585 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent_transition ... ok 2022-11-28T01:32:54.968 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_noncur_transition ... ok 2022-11-28T01:32:54.969 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition ... SKIP 2022-11-28T01:32:54.969 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_multiple_transition ... SKIP 2022-11-28T01:32:54.969 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_noncur_cloud_transition ... SKIP 2022-11-28T01:32:58.611 INFO:teuthology.orchestra.run.smithi003.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition_large_obj ... SKIP 2022-11-28T01:32:58.611 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:32:58.611 INFO:teuthology.orchestra.run.smithi003.stderr:---------------------------------------------------------------------- 2022-11-28T01:32:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:Ran 38 tests in 899.951s 2022-11-28T01:32:58.612 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:32:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:OK (SKIP=4) 2022-11-28T01:32:58.689 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2022-11-28T01:32:58.690 DEBUG:teuthology.orchestra.run.smithi003:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2022-11-28T01:32:58.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-28T01:32:58.713 INFO:tasks.s3tests:Cleaning up boto... 2022-11-28T01:32:58.714 DEBUG:teuthology.orchestra.run.smithi003:> rm /home/ubuntu/cephtest/boto-client.0.cfg 2022-11-28T01:32:58.772 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2022-11-28T01:32:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:32:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:32:58.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.864+0000 7f3d380ce780 20 rados->read ofs=0 len=0 2022-11-28T01:32:58.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.865+0000 7f3d380ce780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:58.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.865+0000 7f3d380ce780 20 realm 2022-11-28T01:32:58.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.865+0000 7f3d380ce780 20 rados->read ofs=0 len=0 2022-11-28T01:32:58.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.865+0000 7f3d380ce780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:58.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.865+0000 7f3d380ce780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:32:58.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.865+0000 7f3d380ce780 20 rados->read ofs=0 len=0 2022-11-28T01:32:58.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.865+0000 7f3d380ce780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:58.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.865+0000 7f3d380ce780 20 rados->read ofs=0 len=0 2022-11-28T01:32:58.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.866+0000 7f3d380ce780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:32:58.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.866+0000 7f3d380ce780 20 rados->read ofs=0 len=0 2022-11-28T01:32:58.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.866+0000 7f3d380ce780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:32:58.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.866+0000 7f3d380ce780 20 searching for the correct realm 2022-11-28T01:32:58.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.873+0000 7f3d380ce780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:32:58.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.873+0000 7f3d380ce780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:32:58.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.873+0000 7f3d380ce780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:32:58.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.873+0000 7f3d380ce780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:32:58.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.873+0000 7f3d380ce780 20 rados->read ofs=0 len=0 2022-11-28T01:32:58.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.873+0000 7f3d380ce780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:58.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.873+0000 7f3d380ce780 20 rados->read ofs=0 len=0 2022-11-28T01:32:58.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:32:58.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 20 rados->read ofs=0 len=0 2022-11-28T01:32:58.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:32:58.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 20 zone default found 2022-11-28T01:32:58.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 4 Realm: () 2022-11-28T01:32:58.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:32:58.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:32:58.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:32:58.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 20 zonegroup default 2022-11-28T01:32:58.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 20 rados->read ofs=0 len=0 2022-11-28T01:32:58.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:58.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 20 rados->read ofs=0 len=0 2022-11-28T01:32:58.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:58.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.874+0000 7f3d380ce780 20 rados->read ofs=0 len=0 2022-11-28T01:32:58.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.875+0000 7f3d380ce780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:58.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.875+0000 7f3d380ce780 20 started sync module instance, tier type = 2022-11-28T01:32:58.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.875+0000 7f3d380ce780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:32:58.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.899+0000 7f3d380ce780 20 add_watcher() i=0 2022-11-28T01:32:58.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.899+0000 7f3d380ce780 20 add_watcher() i=1 2022-11-28T01:32:58.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.899+0000 7f3d380ce780 20 add_watcher() i=2 2022-11-28T01:32:58.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.899+0000 7f3d380ce780 20 add_watcher() i=3 2022-11-28T01:32:58.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.899+0000 7f3d380ce780 20 add_watcher() i=4 2022-11-28T01:32:58.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.899+0000 7f3d380ce780 20 add_watcher() i=5 2022-11-28T01:32:58.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.899+0000 7f3d380ce780 20 add_watcher() i=6 2022-11-28T01:32:58.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.901+0000 7f3d380ce780 20 add_watcher() i=7 2022-11-28T01:32:58.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.901+0000 7f3d380ce780 2 all 8 watchers are set, enabling cache 2022-11-28T01:32:58.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.904+0000 7f3d1d04a700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:32:58.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.904+0000 7f3d380ce780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:32:58.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.904+0000 7f3d380ce780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:32:58.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.959+0000 7f3d18841700 20 reqs_thread_entry: start 2022-11-28T01:32:58.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.959+0000 7f3d380ce780 20 init_complete bucket index max shards: 11 2022-11-28T01:32:58.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.959+0000 7f3d380ce780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:32:58.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.959+0000 7f3d1683d700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:33:29 2022 2022-11-28T01:32:58.963 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:32:58.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.959+0000 7f3d1601b700 20 reqs_thread_entry: start 2022-11-28T01:32:58.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.959+0000 7f3d380ce780 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2022-11-28T01:32:58.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.959+0000 7f3d380ce780 20 rados->read ofs=0 len=0 2022-11-28T01:32:58.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.960+0000 7f3d380ce780 20 rados_obj.operate() r=0 bl.length=469 2022-11-28T01:32:58.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.960+0000 7f3d380ce780 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2022-11-28T01:32:58.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.960+0000 7f3d380ce780 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2022-11-28T01:32:58.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.960+0000 7f3d380ce780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2022-11-28T01:32:58.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.960+0000 7f3d380ce780 10 removing key index: XJYGHSERBPHNMPQLEAZF 2022-11-28T01:32:58.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.960+0000 7f3d380ce780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.keys:XJYGHSERBPHNMPQLEAZF, ofs0, ns] 2022-11-28T01:32:58.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.960+0000 7f3d30871700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659210 cookie 93931970580864 notifier 4332 bl.length()=170 2022-11-28T01:32:58.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.962+0000 7f3d380ce780 10 removing email index: foo.client.0+test@test.test 2022-11-28T01:32:58.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.962+0000 7f3d380ce780 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2022-11-28T01:32:58.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.962+0000 7f3d30070700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659219 cookie 93931970582016 notifier 4332 bl.length()=178 2022-11-28T01:32:58.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.963+0000 7f3d380ce780 10 removing user buckets index 2022-11-28T01:32:58.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.963+0000 7f3d380ce780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2022-11-28T01:32:58.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.963+0000 7f3d30871700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659220 cookie 93931970577408 notifier 4332 bl.length()=169 2022-11-28T01:32:58.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.965+0000 7f3d380ce780 10 removing user index: foo.client.0 2022-11-28T01:32:58.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.965+0000 7f3d380ce780 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2022-11-28T01:32:58.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.965+0000 7f3d380ce780 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2022-11-28T01:32:58.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.965+0000 7f3d30070700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659211 cookie 93931970576256 notifier 4332 bl.length()=161 2022-11-28T01:32:58.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.967+0000 7f3d1d04a700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:32:58.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.969+0000 7f3d380ce780 20 remove_watcher() i=0 2022-11-28T01:32:58.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.969+0000 7f3d380ce780 2 removed watcher, disabling cache 2022-11-28T01:32:58.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.970+0000 7f3d380ce780 20 remove_watcher() i=1 2022-11-28T01:32:58.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.971+0000 7f3d380ce780 20 remove_watcher() i=2 2022-11-28T01:32:58.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.971+0000 7f3d380ce780 20 remove_watcher() i=3 2022-11-28T01:32:58.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.973+0000 7f3d380ce780 20 remove_watcher() i=4 2022-11-28T01:32:58.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.974+0000 7f3d380ce780 20 remove_watcher() i=5 2022-11-28T01:32:58.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.977+0000 7f3d380ce780 20 remove_watcher() i=6 2022-11-28T01:32:58.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:58.978+0000 7f3d380ce780 20 remove_watcher() i=7 2022-11-28T01:32:58.984 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2022-11-28T01:32:59.030 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:32:59.030 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:32:59.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.037+0000 7f95c72ee780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.038+0000 7f95c72ee780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.038+0000 7f95c72ee780 20 realm 2022-11-28T01:32:59.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.038+0000 7f95c72ee780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.038+0000 7f95c72ee780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.038+0000 7f95c72ee780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:32:59.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.038+0000 7f95c72ee780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.038+0000 7f95c72ee780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.038+0000 7f95c72ee780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.039+0000 7f95c72ee780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:32:59.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.039+0000 7f95c72ee780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.039+0000 7f95c72ee780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:32:59.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.039+0000 7f95c72ee780 20 searching for the correct realm 2022-11-28T01:32:59.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.046+0000 7f95c72ee780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:32:59.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.046+0000 7f95c72ee780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:32:59.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.046+0000 7f95c72ee780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:32:59.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.046+0000 7f95c72ee780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:32:59.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.046+0000 7f95c72ee780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.046+0000 7f95c72ee780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.046+0000 7f95c72ee780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.046+0000 7f95c72ee780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:32:59.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.046+0000 7f95c72ee780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:32:59.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 20 zone default found 2022-11-28T01:32:59.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 4 Realm: () 2022-11-28T01:32:59.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:32:59.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:32:59.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:32:59.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 20 zonegroup default 2022-11-28T01:32:59.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 20 started sync module instance, tier type = 2022-11-28T01:32:59.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.047+0000 7f95c72ee780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:32:59.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.072+0000 7f95c72ee780 20 add_watcher() i=0 2022-11-28T01:32:59.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.072+0000 7f95c72ee780 20 add_watcher() i=1 2022-11-28T01:32:59.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.072+0000 7f95c72ee780 20 add_watcher() i=2 2022-11-28T01:32:59.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.072+0000 7f95c72ee780 20 add_watcher() i=3 2022-11-28T01:32:59.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.072+0000 7f95c72ee780 20 add_watcher() i=4 2022-11-28T01:32:59.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.072+0000 7f95c72ee780 20 add_watcher() i=5 2022-11-28T01:32:59.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.072+0000 7f95c72ee780 20 add_watcher() i=6 2022-11-28T01:32:59.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.075+0000 7f95c72ee780 20 add_watcher() i=7 2022-11-28T01:32:59.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.075+0000 7f95c72ee780 2 all 8 watchers are set, enabling cache 2022-11-28T01:32:59.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.078+0000 7f95ac26a700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:32:59.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.078+0000 7f95c72ee780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:32:59.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.078+0000 7f95c72ee780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:32:59.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.128+0000 7f95a7a61700 20 reqs_thread_entry: start 2022-11-28T01:32:59.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.128+0000 7f95c72ee780 20 init_complete bucket index max shards: 11 2022-11-28T01:32:59.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.128+0000 7f95c72ee780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:32:59.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.128+0000 7f95a5a5d700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:33:29 2022 2022-11-28T01:32:59.132 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:32:59.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.128+0000 7f95a523b700 20 reqs_thread_entry: start 2022-11-28T01:32:59.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.128+0000 7f95c72ee780 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2022-11-28T01:32:59.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.128+0000 7f95c72ee780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.128+0000 7f95c72ee780 20 rados_obj.operate() r=0 bl.length=469 2022-11-28T01:32:59.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.128+0000 7f95c72ee780 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2022-11-28T01:32:59.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.128+0000 7f95c72ee780 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2022-11-28T01:32:59.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.128+0000 7f95c72ee780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2022-11-28T01:32:59.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.129+0000 7f95c72ee780 10 removing key index: FLQYKDHVOEAGUSRBWTRZ 2022-11-28T01:32:59.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.129+0000 7f95c72ee780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:FLQYKDHVOEAGUSRBWTRZ, ofs0, ns] 2022-11-28T01:32:59.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.129+0000 7f95bfa91700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659221 cookie 94800574481408 notifier 4340 bl.length()=170 2022-11-28T01:32:59.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.130+0000 7f95c72ee780 10 removing email index: bar.client.0+test@test.test 2022-11-28T01:32:59.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.130+0000 7f95c72ee780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2022-11-28T01:32:59.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.130+0000 7f95bfa91700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659222 cookie 94800574481408 notifier 4340 bl.length()=178 2022-11-28T01:32:59.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.131+0000 7f95c72ee780 10 removing user buckets index 2022-11-28T01:32:59.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.131+0000 7f95c72ee780 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2022-11-28T01:32:59.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.132+0000 7f95bfa91700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659236 cookie 94800574479104 notifier 4340 bl.length()=169 2022-11-28T01:32:59.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.133+0000 7f95c72ee780 10 removing user index: bar.client.0 2022-11-28T01:32:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.133+0000 7f95c72ee780 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2022-11-28T01:32:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.133+0000 7f95c72ee780 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2022-11-28T01:32:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.133+0000 7f95bf290700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659223 cookie 94800574481408 notifier 4340 bl.length()=161 2022-11-28T01:32:59.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.135+0000 7f95ac26a700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:32:59.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.137+0000 7f95c72ee780 20 remove_watcher() i=0 2022-11-28T01:32:59.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.137+0000 7f95c72ee780 2 removed watcher, disabling cache 2022-11-28T01:32:59.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.138+0000 7f95c72ee780 20 remove_watcher() i=1 2022-11-28T01:32:59.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.139+0000 7f95c72ee780 20 remove_watcher() i=2 2022-11-28T01:32:59.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.140+0000 7f95c72ee780 20 remove_watcher() i=3 2022-11-28T01:32:59.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.141+0000 7f95c72ee780 20 remove_watcher() i=4 2022-11-28T01:32:59.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.142+0000 7f95c72ee780 20 remove_watcher() i=5 2022-11-28T01:32:59.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.143+0000 7f95c72ee780 20 remove_watcher() i=6 2022-11-28T01:32:59.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.143+0000 7f95c72ee780 20 remove_watcher() i=7 2022-11-28T01:32:59.149 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2022-11-28T01:32:59.196 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:32:59.197 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:32:59.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.204+0000 7f3e48a74780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.204+0000 7f3e48a74780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.204+0000 7f3e48a74780 20 realm 2022-11-28T01:32:59.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.204+0000 7f3e48a74780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.205+0000 7f3e48a74780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.205+0000 7f3e48a74780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:32:59.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.205+0000 7f3e48a74780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.205+0000 7f3e48a74780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.205+0000 7f3e48a74780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.205+0000 7f3e48a74780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:32:59.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.205+0000 7f3e48a74780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.206+0000 7f3e48a74780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:32:59.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.206+0000 7f3e48a74780 20 searching for the correct realm 2022-11-28T01:32:59.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:32:59.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:32:59.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:32:59.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:32:59.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:32:59.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:32:59.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 zone default found 2022-11-28T01:32:59.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 4 Realm: () 2022-11-28T01:32:59.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:32:59.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:32:59.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:32:59.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 zonegroup default 2022-11-28T01:32:59.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.213+0000 7f3e48a74780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.214+0000 7f3e48a74780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.214+0000 7f3e48a74780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.214+0000 7f3e48a74780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.214+0000 7f3e48a74780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.214+0000 7f3e48a74780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.214+0000 7f3e48a74780 20 started sync module instance, tier type = 2022-11-28T01:32:59.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.214+0000 7f3e48a74780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:32:59.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.225+0000 7f3e48a74780 20 add_watcher() i=0 2022-11-28T01:32:59.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.225+0000 7f3e48a74780 20 add_watcher() i=1 2022-11-28T01:32:59.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.225+0000 7f3e48a74780 20 add_watcher() i=2 2022-11-28T01:32:59.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.225+0000 7f3e48a74780 20 add_watcher() i=3 2022-11-28T01:32:59.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.225+0000 7f3e48a74780 20 add_watcher() i=4 2022-11-28T01:32:59.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.225+0000 7f3e48a74780 20 add_watcher() i=5 2022-11-28T01:32:59.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.225+0000 7f3e48a74780 20 add_watcher() i=6 2022-11-28T01:32:59.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.226+0000 7f3e48a74780 20 add_watcher() i=7 2022-11-28T01:32:59.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.226+0000 7f3e48a74780 2 all 8 watchers are set, enabling cache 2022-11-28T01:32:59.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.229+0000 7f3e2d9f0700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:32:59.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.229+0000 7f3e48a74780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:32:59.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.229+0000 7f3e48a74780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:32:59.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.325+0000 7f3e291e7700 20 reqs_thread_entry: start 2022-11-28T01:32:59.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.325+0000 7f3e48a74780 20 init_complete bucket index max shards: 11 2022-11-28T01:32:59.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.325+0000 7f3e48a74780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:32:59.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.325+0000 7f3e271e3700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:33:29 2022 2022-11-28T01:32:59.328 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:32:59.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.325+0000 7f3e269c1700 20 reqs_thread_entry: start 2022-11-28T01:32:59.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.325+0000 7f3e48a74780 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2022-11-28T01:32:59.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.325+0000 7f3e48a74780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.326+0000 7f3e48a74780 20 rados_obj.operate() r=0 bl.length=528 2022-11-28T01:32:59.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.326+0000 7f3e48a74780 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2022-11-28T01:32:59.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.326+0000 7f3e48a74780 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2022-11-28T01:32:59.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.327+0000 7f3e48a74780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2022-11-28T01:32:59.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.327+0000 7f3e48a74780 10 removing key index: TEOAHZBNNNEQOSSXMKJQ 2022-11-28T01:32:59.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.327+0000 7f3e48a74780 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.keys:TEOAHZBNNNEQOSSXMKJQ, ofs0, ns] 2022-11-28T01:32:59.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.327+0000 7f3e41217700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659224 cookie 94833398314496 notifier 4348 bl.length()=170 2022-11-28T01:32:59.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.329+0000 7f3e48a74780 10 removing email index: testx$tenanteduser.client.0+test@test.test 2022-11-28T01:32:59.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.329+0000 7f3e48a74780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2022-11-28T01:32:59.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.330+0000 7f3e41217700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659212 cookie 94833398315648 notifier 4348 bl.length()=193 2022-11-28T01:32:59.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.331+0000 7f3e48a74780 10 removing user buckets index 2022-11-28T01:32:59.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.331+0000 7f3e48a74780 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2022-11-28T01:32:59.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.331+0000 7f3e40a16700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659225 cookie 94833398314496 notifier 4348 bl.length()=184 2022-11-28T01:32:59.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.333+0000 7f3e48a74780 10 removing user index: testx$tenanteduser.client.0 2022-11-28T01:32:59.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.333+0000 7f3e48a74780 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2022-11-28T01:32:59.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.333+0000 7f3e48a74780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2022-11-28T01:32:59.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.333+0000 7f3e41217700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659213 cookie 94833398313344 notifier 4348 bl.length()=176 2022-11-28T01:32:59.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.336+0000 7f3e2d9f0700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:32:59.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.339+0000 7f3e48a74780 20 remove_watcher() i=0 2022-11-28T01:32:59.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.339+0000 7f3e48a74780 2 removed watcher, disabling cache 2022-11-28T01:32:59.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.340+0000 7f3e48a74780 20 remove_watcher() i=1 2022-11-28T01:32:59.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.342+0000 7f3e48a74780 20 remove_watcher() i=2 2022-11-28T01:32:59.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.343+0000 7f3e48a74780 20 remove_watcher() i=3 2022-11-28T01:32:59.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.345+0000 7f3e48a74780 20 remove_watcher() i=4 2022-11-28T01:32:59.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.347+0000 7f3e48a74780 20 remove_watcher() i=5 2022-11-28T01:32:59.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.349+0000 7f3e48a74780 20 remove_watcher() i=6 2022-11-28T01:32:59.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.350+0000 7f3e48a74780 20 remove_watcher() i=7 2022-11-28T01:32:59.357 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foobar.client.0 --purge-data --cluster ceph 2022-11-28T01:32:59.409 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setuser ceph since I am not root 2022-11-28T01:32:59.409 INFO:teuthology.orchestra.run.smithi003.stderr:ignoring --setgroup ceph since I am not root 2022-11-28T01:32:59.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.418+0000 7f4ccff20780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.419+0000 7f4ccff20780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.419+0000 7f4ccff20780 20 realm 2022-11-28T01:32:59.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.419+0000 7f4ccff20780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.419+0000 7f4ccff20780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.419+0000 7f4ccff20780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-11-28T01:32:59.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.419+0000 7f4ccff20780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.420+0000 7f4ccff20780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.420+0000 7f4ccff20780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.420+0000 7f4ccff20780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:32:59.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.420+0000 7f4ccff20780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.421+0000 7f4ccff20780 20 rados_obj.operate() r=0 bl.length=1003 2022-11-28T01:32:59.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.421+0000 7f4ccff20780 20 searching for the correct realm 2022-11-28T01:32:59.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.428+0000 7f4ccff20780 20 RGWRados::pool_iterate: got zonegroup_info.c3e7c392-6472-46b4-bfd5-1b760e08d894 2022-11-28T01:32:59.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.428+0000 7f4ccff20780 20 RGWRados::pool_iterate: got zone_info.743c3e26-5a1f-4790-8db1-cbe73b4911f7 2022-11-28T01:32:59.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.428+0000 7f4ccff20780 20 RGWRados::pool_iterate: got zone_names.default 2022-11-28T01:32:59.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.428+0000 7f4ccff20780 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-11-28T01:32:59.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.428+0000 7f4ccff20780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.428+0000 7f4ccff20780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.428+0000 7f4ccff20780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 20 rados_obj.operate() r=0 bl.length=46 2022-11-28T01:32:59.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 20 rados_obj.operate() r=0 bl.length=432 2022-11-28T01:32:59.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 20 zone default found 2022-11-28T01:32:59.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 4 Realm: () 2022-11-28T01:32:59.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 4 ZoneGroup: default (c3e7c392-6472-46b4-bfd5-1b760e08d894) 2022-11-28T01:32:59.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 4 Zone: default (743c3e26-5a1f-4790-8db1-cbe73b4911f7) 2022-11-28T01:32:59.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 10 cannot find current period zonegroup using local zonegroup configuration 2022-11-28T01:32:59.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 20 zonegroup default 2022-11-28T01:32:59.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.429+0000 7f4ccff20780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.430+0000 7f4ccff20780 20 rados_obj.operate() r=-2 bl.length=0 2022-11-28T01:32:59.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.430+0000 7f4ccff20780 20 started sync module instance, tier type = 2022-11-28T01:32:59.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.430+0000 7f4ccff20780 20 started zone id=743c3e26-5a1f-4790-8db1-cbe73b4911f7 (name=default) with tier type = 2022-11-28T01:32:59.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.461+0000 7f4ccff20780 20 add_watcher() i=0 2022-11-28T01:32:59.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.461+0000 7f4ccff20780 20 add_watcher() i=1 2022-11-28T01:32:59.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.461+0000 7f4ccff20780 20 add_watcher() i=2 2022-11-28T01:32:59.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.461+0000 7f4ccff20780 20 add_watcher() i=3 2022-11-28T01:32:59.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.461+0000 7f4ccff20780 20 add_watcher() i=4 2022-11-28T01:32:59.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.461+0000 7f4ccff20780 20 add_watcher() i=5 2022-11-28T01:32:59.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.461+0000 7f4ccff20780 20 add_watcher() i=6 2022-11-28T01:32:59.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.462+0000 7f4ccff20780 20 add_watcher() i=7 2022-11-28T01:32:59.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.462+0000 7f4ccff20780 2 all 8 watchers are set, enabling cache 2022-11-28T01:32:59.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.466+0000 7f4cb4e9c700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:32:59.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.466+0000 7f4ccff20780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-11-28T01:32:59.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.466+0000 7f4ccff20780 20 check_secure_mon_conn(): mode 1 is insecure 2022-11-28T01:32:59.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.614+0000 7f4cb0693700 20 reqs_thread_entry: start 2022-11-28T01:32:59.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.614+0000 7f4ccff20780 20 init_complete bucket index max shards: 11 2022-11-28T01:32:59.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.614+0000 7f4ccff20780 10 rgw notify: Started notification manager with: 1 workers 2022-11-28T01:32:59.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.614+0000 7f4cae68f700 20 rgw notify: INFO: next queues processing will happen at: Mon Nov 28 01:33:29 2022 2022-11-28T01:32:59.618 INFO:teuthology.orchestra.run.smithi003.stderr: 2022-11-28T01:32:59.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.614+0000 7f4cade6d700 20 reqs_thread_entry: start 2022-11-28T01:32:59.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.614+0000 7f4ccff20780 10 cache get: name=default.rgw.meta+users.uid+foobar.client.0 : miss 2022-11-28T01:32:59.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.614+0000 7f4ccff20780 20 rados->read ofs=0 len=0 2022-11-28T01:32:59.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.614+0000 7f4ccff20780 20 rados_obj.operate() r=0 bl.length=515 2022-11-28T01:32:59.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.614+0000 7f4ccff20780 10 cache put: name=default.rgw.meta+users.uid+foobar.client.0 info.flags=0x13 2022-11-28T01:32:59.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.614+0000 7f4ccff20780 10 adding default.rgw.meta+users.uid+foobar.client.0 to cache LRU end 2022-11-28T01:32:59.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.614+0000 7f4ccff20780 10 updating xattr: name=ceph.objclass.version bl.length()=42 2022-11-28T01:32:59.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.615+0000 7f4ccff20780 10 removing key index: FQXPRDMECGUAMFUROOWN 2022-11-28T01:32:59.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.615+0000 7f4ccff20780 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.keys:FQXPRDMECGUAMFUROOWN, ofs0, ns] 2022-11-28T01:32:59.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.615+0000 7f4cc86c3700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659237 cookie 93870529808512 notifier 4356 bl.length()=170 2022-11-28T01:32:59.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.618+0000 7f4ccff20780 10 removing email index: foobar.client.0+test@test.test 2022-11-28T01:32:59.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.618+0000 7f4ccff20780 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.email:foobar.client.0+test@test.test, ofs0, ns] 2022-11-28T01:32:59.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.618+0000 7f4cc7ec2700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659214 cookie 93870529810816 notifier 4356 bl.length()=181 2022-11-28T01:32:59.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.623+0000 7f4ccff20780 10 removing user buckets index 2022-11-28T01:32:59.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.623+0000 7f4ccff20780 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foobar.client.0.buckets, ofs0, ns] 2022-11-28T01:32:59.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.623+0000 7f4cc86c3700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659215 cookie 93870529806208 notifier 4356 bl.length()=172 2022-11-28T01:32:59.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.627+0000 7f4ccff20780 10 removing user index: foobar.client.0 2022-11-28T01:32:59.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.627+0000 7f4ccff20780 10 removing default.rgw.meta+users.uid+foobar.client.0 from cache 2022-11-28T01:32:59.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.627+0000 7f4ccff20780 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.uid:foobar.client.0, ofs0, ns] 2022-11-28T01:32:59.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.627+0000 7f4cc7ec2700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 176093659216 cookie 93870529813120 notifier 4356 bl.length()=164 2022-11-28T01:32:59.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.629+0000 7f4cb4e9c700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-11-28T01:32:59.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.636+0000 7f4ccff20780 20 remove_watcher() i=0 2022-11-28T01:32:59.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.636+0000 7f4ccff20780 2 removed watcher, disabling cache 2022-11-28T01:32:59.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.642+0000 7f4ccff20780 20 remove_watcher() i=1 2022-11-28T01:32:59.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.644+0000 7f4ccff20780 20 remove_watcher() i=2 2022-11-28T01:32:59.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.648+0000 7f4ccff20780 20 remove_watcher() i=3 2022-11-28T01:32:59.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.649+0000 7f4ccff20780 20 remove_watcher() i=4 2022-11-28T01:32:59.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.650+0000 7f4ccff20780 20 remove_watcher() i=5 2022-11-28T01:32:59.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.653+0000 7f4ccff20780 20 remove_watcher() i=6 2022-11-28T01:32:59.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-11-28T01:32:59.658+0000 7f4ccff20780 20 remove_watcher() i=7 2022-11-28T01:32:59.664 INFO:tasks.s3tests:Removing s3-tests... 2022-11-28T01:32:59.665 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/s3-tests-client.0 2022-11-28T01:32:59.787 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2022-11-28T01:32:59.798 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2022-11-28T01:32:59.818 DEBUG:tasks.rgw.client.0:waiting for process to exit 2022-11-28T01:32:59.818 INFO:teuthology.orchestra.run:waiting for 300 2022-11-28T01:32:59.819 INFO:tasks.rgw.client.0.smithi003.stdout:2022-11-28T01:32:59.817+0000 7fc9fb23f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground (PID: 111524) UID: 0 2022-11-28T01:32:59.820 INFO:tasks.rgw.client.0.smithi003.stdout:2022-11-28T01:32:59.817+0000 7fc9ff4a87c0 -1 shutting down 2022-11-28T01:33:03.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2022-11-28T01:33:05.920 INFO:tasks.rgw.client.0:Stopped 2022-11-28T01:33:05.921 DEBUG:teuthology.orchestra.run.smithi003:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2022-11-28T01:33:05.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/vault-root-token 2022-11-28T01:33:06.019 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2022-11-28T01:33:06.031 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2022-11-28T01:33:06.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-11-28T01:33:06.390 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-11-28T01:33:06.391 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":41,"fsid":"b763e1e5-4156-4186-b9c8-f7e325a67a93","created":"2022-11-28T01:16:54.587105+0000","modified":"2022-11-28T01:28:12.731899+0000","last_up_change":"2022-11-28T01:17:02.584762+0000","last_in_change":"2022-11-28T01:16:55.558406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-11-28T01:17:02.375638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-11-28T01:17:03.371589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2022-11-28T01:17:09.134133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2022-11-28T01:17:11.408819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2022-11-28T01:17:13.178858+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2022-11-28T01:17:15.112779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2022-11-28T01:17:17.087861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2022-11-28T01:17:19.150956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2022-11-28T01:17:24.225335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2022-11-28T01:26:02.032194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"default.rgw.buckets.data.lukewarm","create_time":"2022-11-28T01:27:09.102056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"default.rgw.buckets.data.frozen","create_time":"2022-11-28T01:28:10.570790+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"08e9c1b7-5217-4527-9277-4a3e76f7f412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":106349},{"type":"v1","addr":"172.21.15.3:6801","nonce":106349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":106349},{"type":"v1","addr":"172.21.15.3:6803","nonce":106349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":106349},{"type":"v1","addr":"172.21.15.3:6807","nonce":106349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":106349},{"type":"v1","addr":"172.21.15.3:6805","nonce":106349}]},"public_addr":"172.21.15.3:6801/106349","cluster_addr":"172.21.15.3:6803/106349","heartbeat_back_addr":"172.21.15.3:6807/106349","heartbeat_front_addr":"172.21.15.3:6805/106349","state":["exists","up"]},{"osd":1,"uuid":"76aed755-6917-448b-836d-664421ae424a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":106351},{"type":"v1","addr":"172.21.15.3:6809","nonce":106351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":106351},{"type":"v1","addr":"172.21.15.3:6811","nonce":106351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":106351},{"type":"v1","addr":"172.21.15.3:6815","nonce":106351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":106351},{"type":"v1","addr":"172.21.15.3:6813","nonce":106351}]},"public_addr":"172.21.15.3:6809/106351","cluster_addr":"172.21.15.3:6811/106351","heartbeat_back_addr":"172.21.15.3:6815/106351","heartbeat_front_addr":"172.21.15.3:6813/106351","state":["exists","up"]},{"osd":2,"uuid":"eeb27915-4b5e-4e0c-b8e6-75a572fd0950","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":106380},{"type":"v1","addr":"172.21.15.3:6817","nonce":106380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":106380},{"type":"v1","addr":"172.21.15.3:6819","nonce":106380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":106380},{"type":"v1","addr":"172.21.15.3:6823","nonce":106380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":106380},{"type":"v1","addr":"172.21.15.3:6821","nonce":106380}]},"public_addr":"172.21.15.3:6817/106380","cluster_addr":"172.21.15.3:6819/106380","heartbeat_back_addr":"172.21.15.3:6823/106380","heartbeat_front_addr":"172.21.15.3:6821/106380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-28T01:16:59.912627+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-28T01:16:59.898210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-28T01:16:59.994384+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.20","mappings":[{"from":0,"to":1}]},{"pgid":"3.33","mappings":[{"from":0,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":0}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]},{"pgid":"4.15","mappings":[{"from":1,"to":2}]},{"pgid":"4.20","mappings":[{"from":1,"to":0}]},{"pgid":"4.35","mappings":[{"from":1,"to":2}]},{"pgid":"4.3a","mappings":[{"from":1,"to":0}]},{"pgid":"10.2","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-11-28T01:33:06.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2022-11-28T01:33:06.744 INFO:teuthology.misc:Shutting down mds daemons... 2022-11-28T01:33:06.745 INFO:teuthology.misc:Shutting down osd daemons... 2022-11-28T01:33:06.746 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2022-11-28T01:33:06.746 INFO:teuthology.orchestra.run:waiting for 300 2022-11-28T01:33:06.780 INFO:tasks.ceph.osd.0:Stopped 2022-11-28T01:33:06.781 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2022-11-28T01:33:06.782 INFO:teuthology.orchestra.run:waiting for 300 2022-11-28T01:33:06.849 INFO:tasks.ceph.osd.1:Stopped 2022-11-28T01:33:06.850 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2022-11-28T01:33:06.850 INFO:teuthology.orchestra.run:waiting for 300 2022-11-28T01:33:06.916 INFO:tasks.ceph.osd.2:Stopped 2022-11-28T01:33:06.917 INFO:teuthology.misc:Shutting down mgr daemons... 2022-11-28T01:33:06.917 DEBUG:tasks.ceph.mgr.0:waiting for process to exit 2022-11-28T01:33:06.918 INFO:teuthology.orchestra.run:waiting for 300 2022-11-28T01:33:06.981 INFO:tasks.ceph.mgr.0:Stopped 2022-11-28T01:33:06.981 INFO:teuthology.misc:Shutting down mon daemons... 2022-11-28T01:33:06.982 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2022-11-28T01:33:06.983 INFO:teuthology.orchestra.run:waiting for 300 2022-11-28T01:33:07.032 INFO:tasks.ceph.mon.a:Stopped 2022-11-28T01:33:07.033 INFO:tasks.ceph:Checking cluster log for badness... 2022-11-28T01:33:07.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2022-11-28T01:33:07.101 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi003.front.sepia.ceph.com 2022-11-28T01:33:07.102 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2022-11-28T01:33:07.442 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi003.front.sepia.ceph.com 2022-11-28T01:33:07.443 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2022-11-28T01:33:07.580 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi003.front.sepia.ceph.com 2022-11-28T01:33:07.581 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2022-11-28T01:33:07.686 INFO:tasks.ceph:Archiving mon data... 2022-11-28T01:33:07.688 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2022-11-23_17:40:47-rgw-main-distro-default-smithi/7089245/data/mon.a.tgz 2022-11-28T01:33:07.688 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-11-28T01:33:07.706 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.7eMb4D1kB5 2022-11-28T01:33:07.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7eMb4D1kB5 2022-11-28T01:33:07.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.7eMb4D1kB5 2022-11-28T01:33:07.954 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.7eMb4D1kB5 is 330KB 2022-11-28T01:33:08.053 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.7eMb4D1kB5 2022-11-28T01:33:08.071 INFO:tasks.ceph:Cleaning ceph cluster... 2022-11-28T01:33:08.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-11-28T01:33:08.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2022-11-28T01:33:08.138 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2022-11-28T01:33:08.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-11-28T01:33:08.206 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-11-28T01:33:08.212 INFO:tasks.ceph:Archiving crash dumps... 2022-11-28T01:33:08.214 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-11-23_17:40:47-rgw-main-distro-default-smithi/7089245/remote/smithi003/crash 2022-11-28T01:33:08.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-11-28T01:33:08.281 INFO:tasks.ceph:Compressing logs... 2022-11-28T01:33:08.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-11-28T01:33:24.886 INFO:tasks.ceph:Archiving logs... 2022-11-28T01:33:24.887 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-11-23_17:40:47-rgw-main-distro-default-smithi/7089245/remote/smithi003/log 2022-11-28T01:33:24.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/log/ceph -- . 2022-11-28T01:33:28.044 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-11-28T01:33:28.058 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-11-28T01:33:28.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-11-28T01:33:32.287 DEBUG:teuthology.run_tasks:Unwinding manager clock 2022-11-28T01:33:32.298 INFO:teuthology.task.clock:Checking final clock skew... 2022-11-28T01:33:32.298 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-11-28T01:33:32.318 INFO:teuthology.orchestra.run.smithi003.stderr:bash: ntpq: command not found 2022-11-28T01:33:32.322 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-11-28T01:33:32.322 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2022-11-28T01:33:32.323 INFO:teuthology.orchestra.run.smithi003.stdout:^- hv01.front.sepia.ceph.com 3 6 377 32 -2611us[-2611us] +/- 102ms 2022-11-28T01:33:32.323 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv02.front.sepia.ceph.com 2 7 377 101 +46us[ +47us] +/- 33ms 2022-11-28T01:33:32.323 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 99 -20us[ -20us] +/- 50ms 2022-11-28T01:33:32.324 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2022-11-28T01:33:32.325 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2022-11-28T01:33:32.336 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2022-11-28T01:33:32.337 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2022-11-28T01:33:32.349 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2022-11-28T01:33:32.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-11-28T01:33:32.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-28T01:33:32.493 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2022-11-28T01:33:32.504 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2022-11-28T01:33:32.515 INFO:teuthology.task.internal:Duration was 1667.624423 seconds 2022-11-28T01:33:32.517 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2022-11-28T01:33:32.528 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2022-11-28T01:33:32.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-11-28T01:33:32.580 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-11-28T01:33:32.874 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2022-11-28T01:33:32.875 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2022-11-28T01:33:32.876 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-11-28T01:33:32.899 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-11-28T01:33:32.900 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-11-28T01:33:32.980 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2022-11-28T01:33:32.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-11-28T01:33:33.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2022-11-28T01:33:33.104 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2022-11-28T01:33:33.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-11-28T01:33:33.133 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2022-11-28T01:33:33.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-11-28T01:33:33.201 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2022-11-28T01:33:33.214 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2022-11-28T01:33:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-28T01:33:33.272 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2022-11-28T01:33:33.283 INFO:teuthology.task.internal:Transferring archived files... 2022-11-28T01:33:33.285 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-11-23_17:40:47-rgw-main-distro-default-smithi/7089245/remote/smithi003 2022-11-28T01:33:33.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-11-28T01:33:33.365 INFO:teuthology.task.internal:Removing archive directory... 2022-11-28T01:33:33.365 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2022-11-28T01:33:33.413 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2022-11-28T01:33:33.426 INFO:teuthology.task.internal:Not uploading archives. 2022-11-28T01:33:33.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2022-11-28T01:33:33.438 INFO:teuthology.task.internal:Tidying up after the test... 2022-11-28T01:33:33.439 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-11-28T01:33:33.471 INFO:teuthology.orchestra.run.smithi003.stdout: 262206 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 01:33 /home/ubuntu/cephtest 2022-11-28T01:33:33.472 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2022-11-28T01:33:33.501 INFO:teuthology.run:Summary data: description: rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests} duration: 1667.6244230270386 flavor: default owner: scheduled_yuriw@teuthology success: true 2022-11-28T01:33:33.501 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-11-28T01:33:38.572 INFO:teuthology.run:pass