2024-01-16T04:21:37.961 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-16T04:21:37.961 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} --verbose --archive /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517318 --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517318/orig.config.yaml 2024-01-16T04:21:37.988 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T04:21:38.073 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517318 branch: wip-yuri5-testing-2024-01-11-1300-pacific description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} email: yweinste@redhat.com first_in_suite: false job_id: '7517318' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri5-testing-2024-01-11-1300-pacific ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption_keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: default sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f workunit: branch: wip-yuri5-testing-2024-01-11-1300-pacific sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9013 sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri5-testing-2024-01-11-1300-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMrCOmXvvC5QExnYAs37aTRoqTzxJQMTjdrxQAm9itOC+nKbW1cyOatQX/3ZlUqvyvCKNjuPlA+30FlZFkyfV9k= tasks: - install: null - ceph: null - rgw: client.0: null - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 force-branch: ceph-pacific vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-15_22:05:22 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-16T04:21:38.073 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa; will attempt to use it 2024-01-16T04:21:38.074 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks 2024-01-16T04:21:38.074 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-16T04:21:38.075 INFO:teuthology.task.internal:Checking packages... 2024-01-16T04:21:38.099 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a70aebef70df6ec2d957347ccbfc484e0573dd5f' 2024-01-16T04:21:38.100 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-16T04:21:38.100 INFO:teuthology.packaging:ref: None 2024-01-16T04:21:38.100 INFO:teuthology.packaging:tag: None 2024-01-16T04:21:38.100 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-01-11-1300-pacific 2024-01-16T04:21:38.100 INFO:teuthology.packaging:sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T04:21:38.100 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri5-testing-2024-01-11-1300-pacific 2024-01-16T04:21:38.238 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-449.ga70aebef 2024-01-16T04:21:38.239 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-16T04:21:38.247 INFO:teuthology.task.internal:no buildpackages task found 2024-01-16T04:21:38.247 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-16T04:21:38.301 INFO:teuthology.task.internal:Saving configuration 2024-01-16T04:21:38.312 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-16T04:21:38.330 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T04:21:38.356 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517318', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 04:16:48.021793', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMrCOmXvvC5QExnYAs37aTRoqTzxJQMTjdrxQAm9itOC+nKbW1cyOatQX/3ZlUqvyvCKNjuPlA+30FlZFkyfV9k='} 2024-01-16T04:21:38.356 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-16T04:21:38.362 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-01-16T04:21:38.362 INFO:teuthology.run_tasks:Running task console_log... 2024-01-16T04:21:38.419 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f530a12aee0>, signals=[15]) 2024-01-16T04:21:38.419 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-16T04:21:38.426 INFO:teuthology.task.internal:Opening connections... 2024-01-16T04:21:38.426 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-01-16T04:21:38.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:21:38.503 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-16T04:21:38.510 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-01-16T04:21:38.532 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-01-16T04:21:38.533 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-01-16T04:21:38.597 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-01-16T04:21:38.597 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="8" 2024-01-16T04:21:38.597 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-01-16T04:21:38.598 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-01-16T04:21:38.598 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="8" 2024-01-16T04:21:38.598 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el8" 2024-01-16T04:21:38.598 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-16T04:21:38.598 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-01-16T04:21:38.598 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-16T04:21:38.598 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-01-16T04:21:38.598 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T04:21:38.598 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-16T04:21:38.598 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T04:21:38.599 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-01-16T04:21:38.617 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-16T04:21:38.625 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-16T04:21:38.647 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-16T04:21:38.647 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-01-16T04:21:38.668 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-16T04:21:38.673 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-16T04:21:38.674 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-01-16T04:21:38.744 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-16T04:21:38.777 INFO:teuthology.run_tasks:Running task kernel... 2024-01-16T04:21:38.791 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-16T04:21:38.791 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-16T04:21:38.791 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T04:21:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:21:38.811 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-01-16T04:21:38.874 INFO:teuthology.orchestra.run.smithi038.stdout:4.18.0-489.el8.x86_64 2024-01-16T04:21:38.875 INFO:teuthology.task.kernel:Running kernel on smithi038: 4.18.0-489.el8.x86_64 2024-01-16T04:21:38.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-01-16T04:21:43.599 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - AppStream 19 MB/s | 35 MB 00:01 2024-01-16T04:21:58.124 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - BaseOS 7.5 MB/s | 58 MB 00:07 2024-01-16T04:22:07.591 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras 28 kB/s | 18 kB 00:00 2024-01-16T04:22:08.133 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2024-01-16T04:22:08.468 INFO:teuthology.orchestra.run.smithi038.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-16T04:22:24.426 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 16 MB 00:15 2024-01-16T04:22:28.567 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 289 kB/s | 24 kB 00:00 2024-01-16T04:22:32.492 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-16T04:22:32.493 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-16T04:22:32.846 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:22:32.847 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:22:32.847 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-01-16T04:22:32.847 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:22:32.847 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:22:32.847 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 4.18.0-532.el8 baseos 10 M 2024-01-16T04:22:32.847 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-01-16T04:22:32.847 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core x86_64 4.18.0-532.el8 baseos 43 M 2024-01-16T04:22:32.847 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules x86_64 4.18.0-532.el8 baseos 36 M 2024-01-16T04:22:32.848 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:22:32.848 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:22:32.848 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:22:32.848 INFO:teuthology.orchestra.run.smithi038.stdout:Install 3 Packages 2024-01-16T04:22:32.848 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:22:32.848 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 90 M 2024-01-16T04:22:32.849 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 96 M 2024-01-16T04:22:32.849 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:22:33.602 INFO:teuthology.orchestra.run.smithi038.stdout:(1/3): kernel-4.18.0-532.el8.x86_64.rpm 18 MB/s | 10 MB 00:00 2024-01-16T04:22:34.577 INFO:teuthology.orchestra.run.smithi038.stdout:(2/3): kernel-modules-4.18.0-532.el8.x86_64.rpm 24 MB/s | 36 MB 00:01 2024-01-16T04:22:35.103 INFO:teuthology.orchestra.run.smithi038.stdout:(3/3): kernel-core-4.18.0-532.el8.x86_64.rpm 21 MB/s | 43 MB 00:02 2024-01-16T04:22:35.103 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:22:35.103 INFO:teuthology.orchestra.run.smithi038.stdout:Total 40 MB/s | 90 MB 00:02 2024-01-16T04:22:35.816 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:22:36.262 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:22:36.262 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:22:37.569 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:22:37.589 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:22:40.516 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:22:41.078 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-16T04:22:44.799 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-16T04:22:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-16T04:22:48.885 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-16T04:22:49.161 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:23:12.448 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:23:12.472 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:23:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:23:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-4.18.0-532.el8.x86_64 1/3 2024-01-16T04:23:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-core-4.18.0-532.el8.x86_64 2/3 2024-01-16T04:23:13.719 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:23:13.719 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:23:13.719 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:23:13.719 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-4.18.0-532.el8.x86_64 kernel-core-4.18.0-532.el8.x86_64 2024-01-16T04:23:13.719 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-4.18.0-532.el8.x86_64 2024-01-16T04:23:13.719 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:23:13.719 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:23:13.859 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-01-16T04:23:14.088 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-16T04:23:14.088 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-16T04:23:14.088 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-532.el8.x86_64 2024-01-16T04:23:14.089 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-16T04:23:14.089 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-16T04:23:14.089 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-16T04:23:14.089 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2024-01-16T04:23:14.089 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-01-16T04:23:14.308 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-16T04:23:14.308 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-16T04:23:14.308 DEBUG:teuthology.orchestra.run.smithi038:> sudo rpm -qi grub2-tools 2024-01-16T04:23:14.342 INFO:teuthology.orchestra.run.smithi038.stdout:Name : grub2-tools 2024-01-16T04:23:14.342 INFO:teuthology.orchestra.run.smithi038.stdout:Epoch : 1 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Version : 2.02 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Release : 129.el8 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: x86_64 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Group : System Environment/Base 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Size : 9265586 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:License : GPLv3+ 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Relocations : (not relocatable) 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Packager : CentOS BuildSystem 2024-01-16T04:23:14.343 INFO:teuthology.orchestra.run.smithi038.stdout:Vendor : CentOS 2024-01-16T04:23:14.344 INFO:teuthology.orchestra.run.smithi038.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T04:23:14.344 INFO:teuthology.orchestra.run.smithi038.stdout:Summary : Support tools for GRUB. 2024-01-16T04:23:14.344 INFO:teuthology.orchestra.run.smithi038.stdout:Description : 2024-01-16T04:23:14.344 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:23:14.344 INFO:teuthology.orchestra.run.smithi038.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T04:23:14.344 INFO:teuthology.orchestra.run.smithi038.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T04:23:14.344 INFO:teuthology.orchestra.run.smithi038.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T04:23:14.344 INFO:teuthology.orchestra.run.smithi038.stdout:hardware devices. 2024-01-16T04:23:14.344 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:23:14.344 INFO:teuthology.orchestra.run.smithi038.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T04:23:14.345 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T04:23:14.345 INFO:teuthology.task.kernel:Updating grub on smithi038 to boot 4.18.0-532.el8.x86_64 2024-01-16T04:23:14.345 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T04:23:16.020 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2024-01-16T04:23:18.509 INFO:teuthology.orchestra.run.smithi038.stderr:done 2024-01-16T04:23:18.511 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-01-16T04:23:18.537 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.8zw2oqenKX 2024-01-16T04:23:18.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8zw2oqenKX 2024-01-16T04:23:18.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.8zw2oqenKX 2024-01-16T04:23:18.757 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.8zw2oqenKX is 6KB 2024-01-16T04:23:18.805 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.8zw2oqenKX 2024-01-16T04:23:18.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T04:23:18.895 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-16T04:23:18.895 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-16T04:23:18.896 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-16T04:23:18.896 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64.conf 2024-01-16T04:23:18.897 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64 2024-01-16T04:23:19.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2024-01-16T04:23:49.095 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-16T04:23:49.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-01-16T04:23:49.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:24:49.156 DEBUG:teuthology.orchestra.remote:timed out 2024-01-16T04:24:58.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-01-16T04:24:58.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:24:58.539 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-01-16T04:24:58.885 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-01-16T04:24:58.886 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-532.el8.x86_64"... 2024-01-16T04:24:58.886 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-01-16T04:24:58.903 INFO:teuthology.orchestra.run.smithi038.stdout:4.18.0-532.el8.x86_64 2024-01-16T04:24:58.903 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-532.el8.x86_64 vs 4.18.0-532.el8.x86_64 2024-01-16T04:24:58.903 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T04:24:58.904 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T04:24:59.904 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-16T04:24:59.905 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T04:24:59.993 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2024-01-16T04:25:00.009 DEBUG:teuthology.parallel:result is None 2024-01-16T04:25:00.009 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-16T04:25:00.017 INFO:teuthology.task.internal:Creating test directory... 2024-01-16T04:25:00.017 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T04:25:00.035 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-16T04:25:00.067 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-16T04:25:00.115 INFO:teuthology.task.internal:Creating archive directory... 2024-01-16T04:25:00.115 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T04:25:00.147 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-16T04:25:00.167 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-16T04:25:00.167 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T04:25:00.240 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T04:25:00.251 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T04:25:00.252 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-16T04:25:00.261 INFO:teuthology.task.internal:Configuring sudo... 2024-01-16T04:25:00.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T04:25:00.288 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-16T04:25:00.301 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-16T04:25:00.301 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T04:25:00.347 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T04:25:00.432 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T04:25:00.483 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T04:25:00.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T04:25:00.611 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:25:00.611 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T04:25:00.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-01-16T04:25:00.822 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T04:25:01.104 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-16T04:25:01.114 INFO:teuthology.task.internal:Starting timer... 2024-01-16T04:25:01.114 INFO:teuthology.run_tasks:Running task pcp... 2024-01-16T04:25:01.203 INFO:teuthology.run_tasks:Running task selinux... 2024-01-16T04:25:01.275 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-01-16T04:25:01.366 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-01-16T04:25:01.367 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-16T04:25:01.368 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-01-16T04:25:01.390 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-01-16T04:25:01.390 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive'} 2024-01-16T04:25:01.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T04:25:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:25:01.459 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-16T04:25:01.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-01-16T04:25:01.526 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-16T04:25:01.537 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-16T04:25:01.558 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-16T04:25:01.559 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-16T04:30:59.531 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com')] 2024-01-16T04:30:59.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-01-16T04:30:59.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:30:59.608 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-01-16T04:30:59.681 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-01-16T04:30:59.681 INFO:teuthology.run_tasks:Running task clock... 2024-01-16T04:30:59.707 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-16T04:30:59.708 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T04:30:59.708 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T04:30:59.765 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T04:30:59.781 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T04:30:59.848 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-01-16T04:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-01-16T04:30:59.875 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T04:30:59.889 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T04:30:59.954 INFO:teuthology.orchestra.run.smithi038.stderr:bash: ntpq: command not found 2024-01-16T04:30:59.957 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T04:30:59.957 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-01-16T04:30:59.958 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:30:59.958 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:30:59.958 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:30:59.958 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:30:59.959 INFO:teuthology.run_tasks:Running task install... 2024-01-16T04:30:59.967 DEBUG:teuthology.task.install:project ceph 2024-01-16T04:30:59.967 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'}} 2024-01-16T04:30:59.967 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'} 2024-01-16T04:30:59.967 INFO:teuthology.task.install:Using flavor: default 2024-01-16T04:30:59.973 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-16T04:30:59.973 INFO:teuthology.task.install:extra packages: [] 2024-01-16T04:30:59.973 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'tag': None, 'wait_for_package': False} 2024-01-16T04:30:59.974 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T04:31:00.196 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-01-11-1300-pacific/a70aebef70df6ec2d957347ccbfc484e0573dd5f/centos/8/flavors/default/ 2024-01-16T04:31:00.196 INFO:teuthology.task.install.rpm:Package version is 16.2.14-449.ga70aebef 2024-01-16T04:31:00.304 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-01-11-1300-pacific/a70aebef70df6ec2d957347ccbfc484e0573dd5f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-01-11-1300-pacific/a70aebef70df6ec2d957347ccbfc484e0573dd5f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-01-11-1300-pacific/a70aebef70df6ec2d957347ccbfc484e0573dd5f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T04:31:00.304 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:31:00.304 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T04:31:00.334 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-16T04:31:00.334 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a70aebef70df6ec2d957347ccbfc484e0573dd5f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T04:31:00.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T04:31:00.485 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T04:31:00.553 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-01-16T04:31:00.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-01-16T04:31:00.835 INFO:teuthology.orchestra.run.smithi038.stdout:76 files removed 2024-01-16T04:31:00.864 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw 2024-01-16T04:31:01.737 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 171 kB/s | 76 kB 00:00 2024-01-16T04:31:02.104 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 47 kB/s | 15 kB 00:00 2024-01-16T04:31:02.463 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 4.4 kB/s | 1.4 kB 00:00 2024-01-16T04:31:04.505 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - AppStream 17 MB/s | 35 MB 00:02 2024-01-16T04:31:12.948 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - BaseOS 36 MB/s | 58 MB 00:01 2024-01-16T04:31:22.290 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras 83 kB/s | 18 kB 00:00 2024-01-16T04:31:22.917 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.9 kB 00:00 2024-01-16T04:31:25.292 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - PowerTools 3.0 MB/s | 6.7 MB 00:02 2024-01-16T04:31:26.885 INFO:teuthology.orchestra.run.smithi038.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2024-01-16T04:31:28.111 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2024-01-16T04:31:32.219 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 409 kB/s | 24 kB 00:00 2024-01-16T04:31:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:31:37.217 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:31:37.217 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-01-16T04:31:37.217 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:31:37.217 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:16.2.14-449.ga70aebef.el8 ceph 12 M 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 3.7 M 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 4.2 M 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:16.2.14-449.ga70aebef.el8 ceph 6.1 M 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:16.2.14-449.ga70aebef.el8 ceph 24 M 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:16.2.14-449.ga70aebef.el8 ceph 26 k 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 825 k 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2024-01-16T04:31:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 494 k 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 3.8 M 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:16.2.14-449.ga70aebef.el8 ceph 45 k 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:16.2.14-449.ga70aebef.el8 ceph 99 k 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:16.2.14-449.ga70aebef.el8 ceph 215 k 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:16.2.14-449.ga70aebef.el8 ceph 389 k 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:16.2.14-449.ga70aebef.el8 ceph 369 k 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:16.2.14-449.ga70aebef.el8 ceph 114 k 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:31:37.219 INFO:teuthology.orchestra.run.smithi038.stdout:Install 21 Packages 2024-01-16T04:31:37.220 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 2 Packages 2024-01-16T04:31:37.220 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:31:37.220 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 58 M 2024-01-16T04:31:37.220 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:31:38.593 INFO:teuthology.orchestra.run.smithi038.stdout:(1/23): ceph-radosgw-16.2.14-449.ga70aebef.el8. 10 MB/s | 12 MB 00:01 2024-01-16T04:31:38.646 INFO:teuthology.orchestra.run.smithi038.stdout:(2/23): ceph-selinux-16.2.14-449.ga70aebef.el8. 493 kB/s | 26 kB 00:00 2024-01-16T04:31:38.813 INFO:teuthology.orchestra.run.smithi038.stdout:(3/23): ceph-base-16.2.14-449.ga70aebef.el8.x86 4.5 MB/s | 6.1 MB 00:01 2024-01-16T04:31:38.846 INFO:teuthology.orchestra.run.smithi038.stdout:(4/23): libcephfs2-16.2.14-449.ga70aebef.el8.x8 4.0 MB/s | 825 kB 00:00 2024-01-16T04:31:38.963 INFO:teuthology.orchestra.run.smithi038.stdout:(5/23): libradosstriper1-16.2.14-449.ga70aebef. 3.2 MB/s | 494 kB 00:00 2024-01-16T04:31:39.047 INFO:teuthology.orchestra.run.smithi038.stdout:(6/23): python3-ceph-argparse-16.2.14-449.ga70a 541 kB/s | 45 kB 00:00 2024-01-16T04:31:39.164 INFO:teuthology.orchestra.run.smithi038.stdout:(7/23): librgw2-16.2.14-449.ga70aebef.el8.x86_6 12 MB/s | 3.8 MB 00:00 2024-01-16T04:31:39.198 INFO:teuthology.orchestra.run.smithi038.stdout:(8/23): python3-ceph-common-16.2.14-449.ga70aeb 655 kB/s | 99 kB 00:00 2024-01-16T04:31:39.223 INFO:teuthology.orchestra.run.smithi038.stdout:(9/23): python3-cephfs-16.2.14-449.ga70aebef.el 3.6 MB/s | 215 kB 00:00 2024-01-16T04:31:39.298 INFO:teuthology.orchestra.run.smithi038.stdout:(10/23): python3-rbd-16.2.14-449.ga70aebef.el8. 4.8 MB/s | 369 kB 00:00 2024-01-16T04:31:39.324 INFO:teuthology.orchestra.run.smithi038.stdout:(11/23): python3-rados-16.2.14-449.ga70aebef.el 3.0 MB/s | 389 kB 00:00 2024-01-16T04:31:39.349 INFO:teuthology.orchestra.run.smithi038.stdout:(12/23): python3-rgw-16.2.14-449.ga70aebef.el8. 2.2 MB/s | 114 kB 00:00 2024-01-16T04:31:40.033 INFO:teuthology.orchestra.run.smithi038.stdout:(13/23): ceph-common-16.2.14-449.ga70aebef.el8. 9.4 MB/s | 24 MB 00:02 2024-01-16T04:31:40.058 INFO:teuthology.orchestra.run.smithi038.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 366 kB/s | 259 kB 00:00 2024-01-16T04:31:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 808 kB/s | 47 kB 00:00 2024-01-16T04:31:40.142 INFO:teuthology.orchestra.run.smithi038.stdout:(16/23): librdkafka-1.6.1-1.el8.x86_64.rpm 827 kB/s | 677 kB 00:00 2024-01-16T04:31:40.376 INFO:teuthology.orchestra.run.smithi038.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 128 kB/s | 44 kB 00:00 2024-01-16T04:31:40.651 INFO:teuthology.orchestra.run.smithi038.stdout:(18/23): leveldb-1.22-1.el8.x86_64.rpm 356 kB/s | 181 kB 00:00 2024-01-16T04:31:40.676 INFO:teuthology.orchestra.run.smithi038.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 547 kB/s | 306 kB 00:00 2024-01-16T04:31:40.735 INFO:teuthology.orchestra.run.smithi038.stdout:(20/23): liboath-2.6.2-3.el8.x86_64.rpm 164 kB/s | 59 kB 00:00 2024-01-16T04:31:40.777 INFO:teuthology.orchestra.run.smithi038.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 599 kB/s | 75 kB 00:00 2024-01-16T04:31:41.094 INFO:teuthology.orchestra.run.smithi038.stdout:(22/23): librbd1-16.2.14-449.ga70aebef.el8.x86_ 12 MB/s | 4.2 MB 00:00 2024-01-16T04:31:41.311 INFO:teuthology.orchestra.run.smithi038.stdout:(23/23): librados2-16.2.14-449.ga70aebef.el8.x8 5.9 MB/s | 3.7 MB 00:00 2024-01-16T04:31:41.313 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:31:41.313 INFO:teuthology.orchestra.run.smithi038.stdout:Total 14 MB/s | 58 MB 00:04 2024-01-16T04:31:41.359 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:31:41.410 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:31:41.411 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:31:41.793 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:31:41.796 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:31:42.937 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:31:43.142 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-01-16T04:31:43.319 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-01-16T04:31:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-01-16T04:31:43.961 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:16.2.14-449.ga70aebef.el8.x86_64 3/25 2024-01-16T04:31:44.054 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.14-449.ga70aebef.el8.x86_64 3/25 2024-01-16T04:31:44.330 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:16.2.14-449.ga70aebef.el8.x86_64 4/25 2024-01-16T04:31:44.494 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 5/25 2024-01-16T04:31:44.957 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 5/25 2024-01-16T04:31:45.119 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 6/25 2024-01-16T04:31:45.183 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 6/25 2024-01-16T04:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2024-01-16T04:31:45.952 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-01-16T04:31:46.102 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:16.2.14-449.ga70aebef.el8.x86_64 9/25 2024-01-16T04:31:46.182 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:16.2.14-449.ga70aebef.el8.x86_64 9/25 2024-01-16T04:31:46.366 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-01-16T04:31:46.517 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:16.2.14-449.ga70aebef.el8. 11/25 2024-01-16T04:31:46.722 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:16.2.14-449.ga70aebef.el8.x86_64 12/25 2024-01-16T04:31:46.931 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:16.2.14-449.ga70aebef.el8.x86_64 13/25 2024-01-16T04:31:47.198 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:16.2.14-449.ga70aebef.el8.x86_64 14/25 2024-01-16T04:31:47.385 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:16.2.14-449.ga70aebef.el8.x86_6 15/25 2024-01-16T04:31:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:16.2.14-449.ga70aebef.el8.x86_6 15/25 2024-01-16T04:31:47.648 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-01-16T04:31:47.845 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-01-16T04:31:48.002 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-01-16T04:31:48.169 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:16.2.14-449.ga70aebef.el8.x8 19/25 2024-01-16T04:31:50.789 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 20/25 2024-01-16T04:31:50.977 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 20/25 2024-01-16T04:31:51.721 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 20/25 2024-01-16T04:31:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 21/25 2024-01-16T04:31:52.154 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 21/25 2024-01-16T04:31:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:16.2.14-449.ga70aebef.el8.x86_64 22/25 2024-01-16T04:32:12.193 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:16.2.14-449.ga70aebef.el8.x86_64 22/25 2024-01-16T04:32:12.193 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-01-16T04:32:12.193 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-01-16T04:32:12.193 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /dev 2024-01-16T04:32:12.193 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /run 2024-01-16T04:32:12.193 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-01-16T04:32:12.193 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-01-16T04:32:12.193 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-01-16T04:32:12.194 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-01-16T04:32:12.194 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /dev 2024-01-16T04:32:12.194 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:16.2.14-449.ga70aebef.el8.x86_64 23/25 2024-01-16T04:32:13.620 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-449.ga70aebef.el8.x86_64 23/25 2024-01-16T04:32:13.621 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-01-16T04:32:13.842 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-01-16T04:32:13.842 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-01-16T04:32:14.587 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-01-16T04:32:14.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 1/25 2024-01-16T04:32:14.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 2/25 2024-01-16T04:32:14.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:16.2.14-449.ga70aebef.el8.x86_64 3/25 2024-01-16T04:32:14.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:16.2.14-449.ga70aebef.el8.x86_64 4/25 2024-01-16T04:32:14.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 5/25 2024-01-16T04:32:14.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:16.2.14-449.ga70aebef.el8.x86_6 6/25 2024-01-16T04:32:14.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:16.2.14-449.ga70aebef.el8.x86_64 7/25 2024-01-16T04:32:14.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:16.2.14-449.ga70aebef.el8. 8/25 2024-01-16T04:32:14.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:16.2.14-449.ga70aebef.el8.x8 9/25 2024-01-16T04:32:14.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:16.2.14-449.ga70aebef.el8.x86_64 10/25 2024-01-16T04:32:14.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:16.2.14-449.ga70aebef.el8.x86_64 11/25 2024-01-16T04:32:14.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:16.2.14-449.ga70aebef.el8.x86_64 12/25 2024-01-16T04:32:14.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:16.2.14-449.ga70aebef.el8.x86_64 13/25 2024-01-16T04:32:14.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-01-16T04:32:14.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-01-16T04:32:14.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-01-16T04:32:14.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2024-01-16T04:32:14.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-01-16T04:32:14.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-01-16T04:32:14.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-01-16T04:32:14.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-01-16T04:32:14.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.14-449.ga70aebef.el8.x86_64 22/25 2024-01-16T04:32:14.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-01-16T04:32:14.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 24/25 2024-01-16T04:32:15.957 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-01-16T04:32:15.957 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:15.957 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-01-16T04:32:15.957 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout: leveldb-1.22-1.el8.x86_64 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.958 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-16T04:32:15.959 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.959 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-16T04:32:15.959 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-16T04:32:15.959 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.959 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.959 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.959 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-16T04:32:15.959 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.959 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.959 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:15.959 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:15.959 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:32:16.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-test 2024-01-16T04:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:45 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:32:18.888 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:32:18.889 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:32:18.889 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-01-16T04:32:18.889 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:32:18.889 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:32:18.889 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:16.2.14-449.ga70aebef.el8 ceph 52 M 2024-01-16T04:32:18.889 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-01-16T04:32:18.889 INFO:teuthology.orchestra.run.smithi038.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-16T04:32:18.889 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:16.2.14-449.ga70aebef.el8 ceph 174 k 2024-01-16T04:32:18.889 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2024-01-16T04:32:18.889 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-16T04:32:18.889 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-16T04:32:18.890 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:18.890 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:32:18.890 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:32:18.890 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-01-16T04:32:18.890 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:18.890 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 53 M 2024-01-16T04:32:18.890 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 196 M 2024-01-16T04:32:18.891 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:32:19.285 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): libcephsqlite-16.2.14-449.ga70aebef.el8. 872 kB/s | 174 kB 00:00 2024-01-16T04:32:19.651 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 359 kB/s | 203 kB 00:00 2024-01-16T04:32:19.893 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 308 kB/s | 187 kB 00:00 2024-01-16T04:32:20.269 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 188 kB/s | 71 kB 00:00 2024-01-16T04:32:20.294 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 503 kB/s | 323 kB 00:00 2024-01-16T04:32:21.594 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): ceph-test-16.2.14-449.ga70aebef.el8.x86_ 21 MB/s | 52 MB 00:02 2024-01-16T04:32:21.596 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:32:21.596 INFO:teuthology.orchestra.run.smithi038.stdout:Total 19 MB/s | 53 MB 00:02 2024-01-16T04:32:21.641 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:32:21.663 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:32:21.664 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:32:22.004 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:32:22.007 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:32:22.512 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:32:22.746 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-16T04:32:22.944 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-16T04:32:23.114 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-16T04:32:23.192 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-16T04:32:23.371 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-16T04:32:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:16.2.14-449.ga70aebef.el8.x86_64 5/6 2024-01-16T04:32:28.447 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:16.2.14-449.ga70aebef.el8.x86_64 5/6 2024-01-16T04:32:28.662 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:16.2.14-449.ga70aebef.el8.x86_64 6/6 2024-01-16T04:32:28.963 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-test-2:16.2.14-449.ga70aebef.el8.x86_64 6/6 2024-01-16T04:32:28.963 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:16.2.14-449.ga70aebef.el8.x86_64 1/6 2024-01-16T04:32:28.963 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:16.2.14-449.ga70aebef.el8.x86_64 2/6 2024-01-16T04:32:28.964 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-16T04:32:28.964 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2024-01-16T04:32:28.964 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-16T04:32:29.737 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-16T04:32:29.737 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:29.737 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:32:29.737 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:29.737 INFO:teuthology.orchestra.run.smithi038.stdout: jq-1.6-8.el8.x86_64 2024-01-16T04:32:29.738 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:29.738 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-16T04:32:29.738 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-16T04:32:29.738 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-16T04:32:29.738 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:29.738 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:32:29.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph 2024-01-16T04:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:58 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:32:32.689 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:32:32.692 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:16.2.14-449.ga70aebef.el8 ceph 6.4 k 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:16.2.14-449.ga70aebef.el8 ceph 2.4 M 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:16.2.14-449.ga70aebef.el8 ceph 1.6 M 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 245 k 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:16.2.14-449.ga70aebef.el8 ceph 5.0 M 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:16.2.14-449.ga70aebef.el8 ceph 18 M 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-16T04:32:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-16T04:32:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout:Install 32 Packages 2024-01-16T04:32:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:32.696 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 31 M 2024-01-16T04:32:32.696 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 101 M 2024-01-16T04:32:32.696 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:32:33.412 INFO:teuthology.orchestra.run.smithi038.stdout:(1/32): ceph-16.2.14-449.ga70aebef.el8.x86_64.r 44 kB/s | 6.4 kB 00:00 2024-01-16T04:32:33.555 INFO:teuthology.orchestra.run.smithi038.stdout:(2/32): ceph-mgr-16.2.14-449.ga70aebef.el8.x86_ 5.6 MB/s | 1.6 MB 00:00 2024-01-16T04:32:34.147 INFO:teuthology.orchestra.run.smithi038.stdout:(3/32): ceph-mon-16.2.14-449.ga70aebef.el8.x86_ 6.8 MB/s | 5.0 MB 00:00 2024-01-16T04:32:34.239 INFO:teuthology.orchestra.run.smithi038.stdout:(4/32): ceph-mgr-modules-core-16.2.14-449.ga70a 2.6 MB/s | 245 kB 00:00 2024-01-16T04:32:34.614 INFO:teuthology.orchestra.run.smithi038.stdout:(5/32): ceph-osd-16.2.14-449.ga70aebef.el8.x86_ 17 MB/s | 18 MB 00:01 2024-01-16T04:32:34.665 INFO:teuthology.orchestra.run.smithi038.stdout:(6/32): ceph-mds-16.2.14-449.ga70aebef.el8.x86_ 1.8 MB/s | 2.4 MB 00:01 2024-01-16T04:32:34.698 INFO:teuthology.orchestra.run.smithi038.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 342 kB/s | 157 kB 00:00 2024-01-16T04:32:34.923 INFO:teuthology.orchestra.run.smithi038.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 126 kB/s | 39 kB 00:00 2024-01-16T04:32:35.065 INFO:teuthology.orchestra.run.smithi038.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 155 kB/s | 57 kB 00:00 2024-01-16T04:32:35.090 INFO:teuthology.orchestra.run.smithi038.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 243 kB/s | 103 kB 00:00 2024-01-16T04:32:35.191 INFO:teuthology.orchestra.run.smithi038.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.9 MB/s | 249 kB 00:00 2024-01-16T04:32:35.233 INFO:teuthology.orchestra.run.smithi038.stdout:(12/32): python3-cryptography-3.2.1-7.el8.x86_6 4.4 MB/s | 641 kB 00:00 2024-01-16T04:32:35.258 INFO:teuthology.orchestra.run.smithi038.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.6 MB/s | 109 kB 00:00 2024-01-16T04:32:35.392 INFO:teuthology.orchestra.run.smithi038.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 275 kB/s | 44 kB 00:00 2024-01-16T04:32:35.517 INFO:teuthology.orchestra.run.smithi038.stdout:(15/32): python3-werkzeug-0.12.2-4.el8.noarch.r 771 kB/s | 457 kB 00:00 2024-01-16T04:32:35.542 INFO:teuthology.orchestra.run.smithi038.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 653 kB/s | 185 kB 00:00 2024-01-16T04:32:35.634 INFO:teuthology.orchestra.run.smithi038.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2024-01-16T04:32:35.709 INFO:teuthology.orchestra.run.smithi038.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 544 kB/s | 173 kB 00:00 2024-01-16T04:32:35.751 INFO:teuthology.orchestra.run.smithi038.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 150 kB/s | 18 kB 00:00 2024-01-16T04:32:35.843 INFO:teuthology.orchestra.run.smithi038.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.2 MB/s | 384 kB 00:00 2024-01-16T04:32:35.910 INFO:teuthology.orchestra.run.smithi038.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 243 kB/s | 49 kB 00:00 2024-01-16T04:32:35.935 INFO:teuthology.orchestra.run.smithi038.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 319 kB/s | 59 kB 00:00 2024-01-16T04:32:36.052 INFO:teuthology.orchestra.run.smithi038.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 110 kB/s | 16 kB 00:00 2024-01-16T04:32:36.078 INFO:teuthology.orchestra.run.smithi038.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 132 kB/s | 19 kB 00:00 2024-01-16T04:32:36.187 INFO:teuthology.orchestra.run.smithi038.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 826 kB/s | 283 kB 00:00 2024-01-16T04:32:36.220 INFO:teuthology.orchestra.run.smithi038.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 143 kB/s | 24 kB 00:00 2024-01-16T04:32:36.245 INFO:teuthology.orchestra.run.smithi038.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 174 kB/s | 29 kB 00:00 2024-01-16T04:32:36.304 INFO:teuthology.orchestra.run.smithi038.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 235 kB/s | 27 kB 00:00 2024-01-16T04:32:36.529 INFO:teuthology.orchestra.run.smithi038.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 379 kB/s | 85 kB 00:00 2024-01-16T04:32:36.555 INFO:teuthology.orchestra.run.smithi038.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 765 kB/s | 255 kB 00:00 2024-01-16T04:32:36.621 INFO:teuthology.orchestra.run.smithi038.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 669 kB/s | 251 kB 00:00 2024-01-16T04:32:36.671 INFO:teuthology.orchestra.run.smithi038.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 162 kB/s | 23 kB 00:00 2024-01-16T04:32:36.674 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:32:36.675 INFO:teuthology.orchestra.run.smithi038.stdout:Total 7.9 MB/s | 31 MB 00:03 2024-01-16T04:32:36.760 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:32:36.801 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:32:36.802 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:32:37.120 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:32:37.123 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:32:38.303 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:32:38.521 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-16T04:32:38.729 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-16T04:32:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-16T04:32:39.210 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-16T04:32:39.433 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-16T04:32:39.591 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-16T04:32:39.782 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-16T04:32:39.977 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-16T04:32:40.214 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-16T04:32:40.467 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-16T04:32:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-16T04:32:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-16T04:32:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-16T04:32:41.657 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-16T04:32:41.862 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-16T04:32:42.075 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-16T04:32:42.306 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-16T04:32:42.642 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-16T04:32:42.838 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-16T04:32:43.050 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-16T04:32:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-16T04:32:43.546 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-16T04:32:43.755 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-16T04:32:44.008 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-16T04:32:44.308 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-16T04:32:44.605 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-16T04:32:44.938 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-449.ga70aebef.el8. 27/32 2024-01-16T04:32:45.085 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 28/32 2024-01-16T04:32:46.701 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 28/32 2024-01-16T04:32:47.169 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:16.2.14-449.ga70aebef.el8.x86_64 29/32 2024-01-16T04:32:47.623 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:16.2.14-449.ga70aebef.el8.x86_64 29/32 2024-01-16T04:32:47.836 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:16.2.14-449.ga70aebef.el8.x86_64 30/32 2024-01-16T04:32:48.065 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:16.2.14-449.ga70aebef.el8.x86_64 30/32 2024-01-16T04:32:48.234 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:16.2.14-449.ga70aebef.el8.x86_64 31/32 2024-01-16T04:32:48.253 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:16.2.14-449.ga70aebef.el8.x86_64 31/32 2024-01-16T04:32:48.377 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:16.2.14-449.ga70aebef.el8.x86_64 32/32 2024-01-16T04:32:48.821 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-2:16.2.14-449.ga70aebef.el8.x86_64 32/32 2024-01-16T04:32:48.821 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:16.2.14-449.ga70aebef.el8.x86_64 1/32 2024-01-16T04:32:48.821 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:16.2.14-449.ga70aebef.el8.x86_64 2/32 2024-01-16T04:32:48.821 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 3/32 2024-01-16T04:32:48.821 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:16.2.14-449.ga70aebef.el8.x86_64 4/32 2024-01-16T04:32:48.821 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:16.2.14-449.ga70aebef.el8.x86_64 5/32 2024-01-16T04:32:48.821 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-449.ga70aebef.el8. 6/32 2024-01-16T04:32:48.821 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-16T04:32:48.821 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-16T04:32:48.821 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-16T04:32:48.821 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-16T04:32:48.821 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-16T04:32:48.822 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-16T04:32:48.823 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-16T04:32:48.823 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-16T04:32:48.823 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-16T04:32:48.823 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-16T04:32:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-16T04:32:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:50.014 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:32:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:32:50.015 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:50.015 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:32:50.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-16T04:32:50.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-16T04:32:50.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-16T04:32:50.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-16T04:32:50.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-16T04:32:50.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-16T04:32:50.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-2.6-1.el8.noarch 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-16T04:32:50.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-16T04:32:50.017 INFO:teuthology.orchestra.run.smithi038.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-16T04:32:50.017 INFO:teuthology.orchestra.run.smithi038.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-16T04:32:50.017 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-16T04:32:50.017 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-16T04:32:50.017 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-16T04:32:50.017 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-16T04:32:50.017 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:50.017 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:32:50.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-base 2024-01-16T04:32:51.158 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:19 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:32:52.746 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:32:52.997 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:32:52.999 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-01-16T04:32:52.999 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:32:53.114 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install cephadm 2024-01-16T04:32:54.005 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:22 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:32:55.826 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:32:55.827 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:32:55.827 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-01-16T04:32:55.827 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:32:55.827 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:32:55.827 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 85 k 2024-01-16T04:32:55.827 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:55.827 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:32:55.827 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:32:55.827 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-01-16T04:32:55.827 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:55.828 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 85 k 2024-01-16T04:32:55.828 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 352 k 2024-01-16T04:32:55.828 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:32:55.996 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm-16.2.14-449.ga70aebef.el8.noarch.rpm 509 kB/s | 85 kB 00:00 2024-01-16T04:32:55.996 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:32:55.996 INFO:teuthology.orchestra.run.smithi038.stdout:Total 505 kB/s | 85 kB 00:00 2024-01-16T04:32:55.996 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:32:56.004 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:32:56.004 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:32:56.006 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:32:56.010 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:32:56.079 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:32:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/1 2024-01-16T04:32:56.713 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/1 2024-01-16T04:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/1 2024-01-16T04:32:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/1 2024-01-16T04:32:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:57.659 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:32:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:32:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:32:57.659 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:32:59.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-immutable-object-cache 2024-01-16T04:33:00.481 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:28 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:33:02.334 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:33:02.335 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:02.335 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-01-16T04:33:02.335 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:02.335 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:33:02.335 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-449.ga70aebef.el8 ceph 161 k 2024-01-16T04:33:02.335 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:02.335 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:33:02.336 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:02.336 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-01-16T04:33:02.336 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:02.336 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 161 k 2024-01-16T04:33:02.336 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 470 k 2024-01-16T04:33:02.336 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:33:02.626 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-immutable-object-cache-16.2.14-449.ga70aeb 555 kB/s | 161 kB 00:00 2024-01-16T04:33:02.627 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:33:02.627 INFO:teuthology.orchestra.run.smithi038.stdout:Total 553 kB/s | 161 kB 00:00 2024-01-16T04:33:02.627 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:33:02.640 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:33:02.640 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:33:02.690 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:33:02.693 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:33:02.811 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:33:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-449.ga70aebef. 1/1 2024-01-16T04:33:03.441 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-449.ga70aebef. 1/1 2024-01-16T04:33:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-449.ga70aebef. 1/1 2024-01-16T04:33:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:04.201 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:33:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:33:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:04.201 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:33:04.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr 2024-01-16T04:33:05.299 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:33 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:33:06.878 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:33:07.131 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:33:07.132 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-01-16T04:33:07.132 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:33:07.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-dashboard 2024-01-16T04:33:08.125 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:36 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:33:09.963 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:33:09.964 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:09.964 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-01-16T04:33:09.964 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:09.964 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:33:09.964 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 3.1 M 2024-01-16T04:33:09.965 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-01-16T04:33:09.965 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 24 k 2024-01-16T04:33:09.965 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 15 k 2024-01-16T04:33:09.965 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-16T04:33:09.965 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-16T04:33:09.965 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-16T04:33:09.965 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:09.965 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:33:09.965 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:09.965 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-01-16T04:33:09.965 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:09.966 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.4 M 2024-01-16T04:33:09.966 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 22 M 2024-01-16T04:33:09.966 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:33:10.441 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): ceph-grafana-dashboards-16.2.14-449.ga70 60 kB/s | 24 kB 00:00 2024-01-16T04:33:10.471 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): ceph-prometheus-alerts-16.2.14-449.ga70a 34 kB/s | 15 kB 00:00 2024-01-16T04:33:10.538 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 441 kB/s | 43 kB 00:00 2024-01-16T04:33:10.597 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): ceph-mgr-dashboard-16.2.14-449.ga70aebef 5.6 MB/s | 3.1 MB 00:00 2024-01-16T04:33:10.705 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 144 kB/s | 34 kB 00:00 2024-01-16T04:33:10.747 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 942 kB/s | 196 kB 00:00 2024-01-16T04:33:10.751 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:33:10.751 INFO:teuthology.orchestra.run.smithi038.stdout:Total 4.4 MB/s | 3.4 MB 00:00 2024-01-16T04:33:10.764 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:33:10.776 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:33:10.777 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:33:10.937 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:33:10.941 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:33:11.255 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:33:11.441 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-16T04:33:11.677 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-16T04:33:11.893 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-16T04:33:12.007 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-449.ga70aebef.el8.n 4/6 2024-01-16T04:33:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-449.ga70aebef.el8. 5/6 2024-01-16T04:33:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-449.ga70aebef.el8.noarc 6/6 2024-01-16T04:33:13.660 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-449.ga70aebef.el8.noarc 6/6 2024-01-16T04:33:13.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-449.ga70aebef.el8. 1/6 2024-01-16T04:33:13.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-449.ga70aebef.el8.noarc 2/6 2024-01-16T04:33:13.660 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-449.ga70aebef.el8.n 3/6 2024-01-16T04:33:13.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-16T04:33:13.661 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-16T04:33:14.416 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-16T04:33:14.416 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:14.416 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:33:14.416 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:33:14.416 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:33:14.416 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:33:14.417 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-16T04:33:14.417 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-16T04:33:14.417 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-16T04:33:14.417 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:14.417 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:33:14.553 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-16T04:33:15.446 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:43 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:33:17.286 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:33:17.287 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2024-01-16T04:33:17.287 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-01-16T04:33:17.287 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2024-01-16T04:33:17.287 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:33:17.287 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 8.6 M 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout:Install 8 Packages 2024-01-16T04:33:17.288 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:17.289 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 37 M 2024-01-16T04:33:17.289 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 210 M 2024-01-16T04:33:17.289 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:33:18.154 INFO:teuthology.orchestra.run.smithi038.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-44 14 MB/s | 8.6 MB 00:00 2024-01-16T04:33:18.545 INFO:teuthology.orchestra.run.smithi038.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 4.6 MB/s | 4.7 MB 00:01 2024-01-16T04:33:18.603 INFO:teuthology.orchestra.run.smithi038.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.5 MB/s | 4.9 MB 00:01 2024-01-16T04:33:19.062 INFO:teuthology.orchestra.run.smithi038.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.1 MB/s | 3.7 MB 00:00 2024-01-16T04:33:19.087 INFO:teuthology.orchestra.run.smithi038.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 415 kB/s | 225 kB 00:00 2024-01-16T04:33:19.208 INFO:teuthology.orchestra.run.smithi038.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.4 MB/s | 172 kB 00:00 2024-01-16T04:33:19.263 INFO:teuthology.orchestra.run.smithi038.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.1 MB/s | 645 kB 00:00 2024-01-16T04:33:19.830 INFO:teuthology.orchestra.run.smithi038.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 12 MB/s | 14 MB 00:01 2024-01-16T04:33:19.831 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:33:19.831 INFO:teuthology.orchestra.run.smithi038.stdout:Total 15 MB/s | 37 MB 00:02 2024-01-16T04:33:20.072 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:33:20.091 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:33:20.091 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:33:20.441 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:33:20.444 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:33:21.045 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:33:21.189 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-16T04:33:21.333 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-16T04:33:21.873 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-16T04:33:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-16T04:33:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-16T04:33:23.372 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-16T04:33:23.522 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-16T04:33:24.266 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-16T04:33:24.673 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-16T04:33:25.056 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-16T04:33:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-16T04:33:28.477 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-16T04:33:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-449.ga70aebe 8/8 2024-01-16T04:33:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-449.ga70aebe 8/8 2024-01-16T04:33:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-449.ga70aebe 1/8 2024-01-16T04:33:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-16T04:33:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-16T04:33:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-16T04:33:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-16T04:33:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-16T04:33:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-16T04:33:30.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-16T04:33:30.020 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:30.020 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:33:30.020 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:33:30.020 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-16T04:33:30.020 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-16T04:33:30.020 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-16T04:33:30.020 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-16T04:33:30.020 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-16T04:33:30.020 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-16T04:33:30.020 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-16T04:33:30.020 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:30.020 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:33:30.179 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-rook 2024-01-16T04:33:31.064 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:59 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:33:32.887 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:33:32.889 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:32.889 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-01-16T04:33:32.889 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:32.889 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:33:32.890 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 27 k 2024-01-16T04:33:32.890 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-01-16T04:33:32.890 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-16T04:33:32.890 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-16T04:33:32.890 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-16T04:33:32.890 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-16T04:33:32.890 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-16T04:33:32.890 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-16T04:33:32.890 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-16T04:33:32.890 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-16T04:33:32.890 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-16T04:33:32.890 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-16T04:33:32.891 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-16T04:33:32.891 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-16T04:33:32.891 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:32.891 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:33:32.891 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:32.891 INFO:teuthology.orchestra.run.smithi038.stdout:Install 13 Packages 2024-01-16T04:33:32.891 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:32.892 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 1.9 M 2024-01-16T04:33:32.892 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 24 M 2024-01-16T04:33:32.892 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:33:33.236 INFO:teuthology.orchestra.run.smithi038.stdout:(1/13): ceph-mgr-rook-16.2.14-449.ga70aebef.el8 187 kB/s | 27 kB 00:00 2024-01-16T04:33:33.345 INFO:teuthology.orchestra.run.smithi038.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 78 kB/s | 20 kB 00:00 2024-01-16T04:33:33.370 INFO:teuthology.orchestra.run.smithi038.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 95 kB/s | 27 kB 00:00 2024-01-16T04:33:33.462 INFO:teuthology.orchestra.run.smithi038.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.6 MB/s | 155 kB 00:00 2024-01-16T04:33:33.495 INFO:teuthology.orchestra.run.smithi038.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.2 MB/s | 43 kB 00:00 2024-01-16T04:33:33.646 INFO:teuthology.orchestra.run.smithi038.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 220 kB/s | 33 kB 00:00 2024-01-16T04:33:33.671 INFO:teuthology.orchestra.run.smithi038.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 291 kB/s | 126 kB 00:00 2024-01-16T04:33:33.713 INFO:teuthology.orchestra.run.smithi038.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 244 kB/s | 16 kB 00:00 2024-01-16T04:33:33.797 INFO:teuthology.orchestra.run.smithi038.stdout:(9/13): python3-pyasn1-modules-0.3.7-6.el8.noar 243 kB/s | 110 kB 00:00 2024-01-16T04:33:33.914 INFO:teuthology.orchestra.run.smithi038.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 335 kB/s | 81 kB 00:00 2024-01-16T04:33:34.022 INFO:teuthology.orchestra.run.smithi038.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 566 kB/s | 61 kB 00:00 2024-01-16T04:33:34.081 INFO:teuthology.orchestra.run.smithi038.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 3.2 MB/s | 1.2 MB 00:00 2024-01-16T04:33:34.106 INFO:teuthology.orchestra.run.smithi038.stdout:(13/13): python3-rsa-4.9-2.el8.noarch.rpm 208 kB/s | 64 kB 00:00 2024-01-16T04:33:34.106 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:33:34.106 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.6 MB/s | 1.9 MB 00:01 2024-01-16T04:33:34.172 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:33:34.186 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:33:34.186 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:33:34.279 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:33:34.282 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:33:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-16T04:33:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-16T04:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-16T04:33:35.729 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-16T04:33:35.929 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-16T04:33:36.173 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-16T04:33:36.496 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-16T04:33:36.755 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-16T04:33:37.852 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-16T04:33:38.128 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-16T04:33:38.329 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-16T04:33:38.529 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-16T04:33:38.679 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:16.2.14-449.ga70aebef.el8.noarch 13/13 2024-01-16T04:33:38.998 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-449.ga70aebef.el8.noarch 13/13 2024-01-16T04:33:38.998 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:16.2.14-449.ga70aebef.el8.noarch 1/13 2024-01-16T04:33:38.998 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-16T04:33:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-16T04:33:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-16T04:33:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-16T04:33:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-16T04:33:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-16T04:33:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-16T04:33:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-16T04:33:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-16T04:33:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-16T04:33:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-16T04:33:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-16T04:33:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:39.840 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-16T04:33:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-16T04:33:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:39.842 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:33:39.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-cephadm 2024-01-16T04:33:40.858 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:08 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:33:42.698 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:33:42.699 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-01-16T04:33:42.699 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:42.699 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:33:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 116 k 2024-01-16T04:33:42.699 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-01-16T04:33:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-16T04:33:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-16T04:33:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-16T04:33:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-16T04:33:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-16T04:33:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:42.700 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:33:42.700 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:42.700 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-01-16T04:33:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:42.700 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 5.8 M 2024-01-16T04:33:42.701 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 24 M 2024-01-16T04:33:42.701 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:33:43.074 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): ceph-mgr-cephadm-16.2.14-449.ga70aebef.e 597 kB/s | 116 kB 00:00 2024-01-16T04:33:43.341 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 79 kB/s | 21 kB 00:00 2024-01-16T04:33:43.491 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 879 kB/s | 538 kB 00:00 2024-01-16T04:33:43.733 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 907 kB/s | 355 kB 00:00 2024-01-16T04:33:43.808 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.2 MB/s | 4.8 MB 00:00 2024-01-16T04:33:43.833 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-remoto-1.2.1-1.el8.noarch.rpm 143 kB/s | 49 kB 00:00 2024-01-16T04:33:43.834 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:33:43.834 INFO:teuthology.orchestra.run.smithi038.stdout:Total 5.1 MB/s | 5.8 MB 00:01 2024-01-16T04:33:43.890 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:33:43.901 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:33:43.901 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:33:44.018 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:33:44.021 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:33:44.370 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:33:44.626 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-16T04:33:44.908 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-16T04:33:45.940 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-16T04:33:46.342 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-16T04:33:46.614 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-16T04:33:46.794 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-449.ga70aebef.el8.noarch 6/6 2024-01-16T04:33:47.101 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-449.ga70aebef.el8.noarch 6/6 2024-01-16T04:33:47.101 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/6 2024-01-16T04:33:47.101 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-16T04:33:47.101 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-16T04:33:47.101 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-16T04:33:47.102 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-16T04:33:47.878 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-16T04:33:47.879 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:47.879 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:33:47.879 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:33:47.879 INFO:teuthology.orchestra.run.smithi038.stdout: python3-apipkg-1.5-6.el8.noarch 2024-01-16T04:33:47.879 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-16T04:33:47.879 INFO:teuthology.orchestra.run.smithi038.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-01-16T04:33:47.879 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-16T04:33:47.879 INFO:teuthology.orchestra.run.smithi038.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-01-16T04:33:47.879 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:47.879 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:33:47.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-fuse 2024-01-16T04:33:48.889 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:16 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:33:50.730 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:33:50.731 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:50.731 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-01-16T04:33:50.731 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:50.731 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:33:50.731 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:16.2.14-449.ga70aebef.el8 ceph 911 k 2024-01-16T04:33:50.731 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-01-16T04:33:50.732 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-16T04:33:50.732 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:50.732 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:33:50.732 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:50.732 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2024-01-16T04:33:50.732 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:50.732 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 994 k 2024-01-16T04:33:50.733 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 2.8 M 2024-01-16T04:33:50.733 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:33:51.018 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 891 kB/s | 83 kB 00:00 2024-01-16T04:33:51.343 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): ceph-fuse-16.2.14-449.ga70aebef.el8.x86_ 2.1 MB/s | 911 kB 00:00 2024-01-16T04:33:51.343 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:33:51.344 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.6 MB/s | 994 kB 00:00 2024-01-16T04:33:51.355 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:33:51.377 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:33:51.377 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:33:51.471 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:33:51.475 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:33:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:33:51.935 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-16T04:33:52.186 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 2/2 2024-01-16T04:33:52.602 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 2/2 2024-01-16T04:33:52.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 1/2 2024-01-16T04:33:53.355 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-16T04:33:53.355 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:53.355 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:33:53.356 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-16T04:33:53.356 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:53.356 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:33:54.458 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados-devel 2024-01-16T04:33:55.362 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:23 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:33:57.196 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:33:57.197 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-01-16T04:33:57.197 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:57.197 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:33:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:16.2.14-449.ga70aebef.el8 ceph 131 k 2024-01-16T04:33:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:57.197 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:33:57.197 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:33:57.197 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-01-16T04:33:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:57.198 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 131 k 2024-01-16T04:33:57.198 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 432 k 2024-01-16T04:33:57.198 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:33:57.398 INFO:teuthology.orchestra.run.smithi038.stdout:librados-devel-16.2.14-449.ga70aebef.el8.x86_64 655 kB/s | 131 kB 00:00 2024-01-16T04:33:57.399 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:33:57.399 INFO:teuthology.orchestra.run.smithi038.stdout:Total 650 kB/s | 131 kB 00:00 2024-01-16T04:33:57.399 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:33:57.405 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:33:57.405 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:33:57.458 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:33:57.461 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:33:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:33:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:33:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:33:58.860 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:33:58.860 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:58.861 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:33:58.861 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:33:58.861 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:33:58.861 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:33:59.129 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs2 2024-01-16T04:34:00.029 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:28 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:34:01.601 INFO:teuthology.orchestra.run.smithi038.stdout:Package libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:34:01.850 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:34:01.852 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-01-16T04:34:01.852 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:34:01.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs-devel 2024-01-16T04:34:02.862 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:30 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:34:04.735 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:34:04.736 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:34:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-01-16T04:34:04.736 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:34:04.736 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:34:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:16.2.14-449.ga70aebef.el8 ceph 25 k 2024-01-16T04:34:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:04.736 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:34:04.736 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:34:04.736 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-01-16T04:34:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:04.737 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 25 k 2024-01-16T04:34:04.737 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 107 k 2024-01-16T04:34:04.737 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:34:04.880 INFO:teuthology.orchestra.run.smithi038.stdout:libcephfs-devel-16.2.14-449.ga70aebef.el8.x86_6 178 kB/s | 25 kB 00:00 2024-01-16T04:34:04.881 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:34:04.881 INFO:teuthology.orchestra.run.smithi038.stdout:Total 176 kB/s | 25 kB 00:00 2024-01-16T04:34:04.881 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:34:04.885 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:34:04.885 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:34:04.887 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:34:04.891 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:34:04.939 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:34:05.149 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:34:05.423 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:34:06.131 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:34:06.131 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:06.131 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:34:06.131 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:06.131 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:06.131 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:34:06.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados2 2024-01-16T04:34:07.170 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:35 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:34:08.881 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:34:09.137 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:34:09.138 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-01-16T04:34:09.139 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:34:09.231 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librbd1 2024-01-16T04:34:10.147 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:38 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:34:11.763 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:34:12.017 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:34:12.018 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-01-16T04:34:12.018 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:34:12.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rados 2024-01-16T04:34:13.002 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:41 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:34:14.587 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rados-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:34:14.837 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:34:14.839 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-01-16T04:34:14.839 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:34:14.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rgw 2024-01-16T04:34:15.826 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:43 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:34:17.398 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rgw-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:34:17.653 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:34:17.655 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-01-16T04:34:17.655 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:34:17.761 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-cephfs 2024-01-16T04:34:18.660 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:46 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:34:20.247 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-cephfs-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:34:20.500 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:34:20.501 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-01-16T04:34:20.501 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:34:20.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rbd 2024-01-16T04:34:21.516 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:49 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:34:23.093 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rbd-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:34:23.343 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:34:23.345 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-01-16T04:34:23.345 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:34:23.444 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-fuse 2024-01-16T04:34:24.341 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:52 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:34:26.180 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:34:26.181 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:34:26.181 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-01-16T04:34:26.181 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:34:26.181 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:34:26.181 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:16.2.14-449.ga70aebef.el8 ceph 88 k 2024-01-16T04:34:26.181 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:26.181 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:34:26.181 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:34:26.181 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-01-16T04:34:26.181 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:26.182 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 88 k 2024-01-16T04:34:26.182 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 240 k 2024-01-16T04:34:26.182 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:34:26.485 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-fuse-16.2.14-449.ga70aebef.el8.x86_64.rpm 292 kB/s | 88 kB 00:00 2024-01-16T04:34:26.486 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:34:26.486 INFO:teuthology.orchestra.run.smithi038.stdout:Total 290 kB/s | 88 kB 00:00 2024-01-16T04:34:26.486 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:34:26.497 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:34:26.497 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:34:26.551 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:34:26.554 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:34:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:34:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:34:27.216 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:34:28.046 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:34:28.046 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:28.046 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:34:28.046 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:28.046 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:28.046 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:34:28.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-mirror 2024-01-16T04:34:29.165 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:57 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:34:30.990 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:34:30.991 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:34:30.991 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-01-16T04:34:30.991 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:34:30.991 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:34:30.991 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:16.2.14-449.ga70aebef.el8 ceph 3.7 M 2024-01-16T04:34:30.991 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:30.991 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:34:30.991 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:34:30.991 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-01-16T04:34:30.991 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:30.991 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.7 M 2024-01-16T04:34:30.991 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 15 M 2024-01-16T04:34:30.992 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:34:31.474 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-mirror-16.2.14-449.ga70aebef.el8.x86_64.rpm 7.6 MB/s | 3.7 MB 00:00 2024-01-16T04:34:31.474 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:34:31.475 INFO:teuthology.orchestra.run.smithi038.stdout:Total 7.6 MB/s | 3.7 MB 00:00 2024-01-16T04:34:31.475 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:34:31.488 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:34:31.488 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:34:31.555 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:34:31.558 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:34:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:34:32.289 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:34:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:34:33.432 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:34:33.432 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:33.432 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:34:33.433 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:33.433 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:33.433 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:34:33.638 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-nbd 2024-01-16T04:34:34.539 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:03:02 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:34:36.361 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:34:36.362 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:34:36.362 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-01-16T04:34:36.362 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:34:36.362 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:34:36.362 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:16.2.14-449.ga70aebef.el8 ceph 175 k 2024-01-16T04:34:36.362 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:36.362 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:34:36.362 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:34:36.362 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-01-16T04:34:36.362 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:36.362 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 175 k 2024-01-16T04:34:36.362 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 523 k 2024-01-16T04:34:36.363 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:34:36.660 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-nbd-16.2.14-449.ga70aebef.el8.x86_64.rpm 588 kB/s | 175 kB 00:00 2024-01-16T04:34:36.661 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:34:36.661 INFO:teuthology.orchestra.run.smithi038.stdout:Total 585 kB/s | 175 kB 00:00 2024-01-16T04:34:36.661 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:34:36.673 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:34:36.673 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:34:36.727 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:34:36.731 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:34:36.849 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:34:37.062 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:34:37.355 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-nbd-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:34:38.142 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:34:38.142 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:38.142 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:34:38.142 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:38.142 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:38.142 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:34:38.343 DEBUG:teuthology.parallel:result is None 2024-01-16T04:34:38.344 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T04:34:38.574 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T04:34:38.597 INFO:teuthology.orchestra.run.smithi038.stdout:16.2.14-449.ga70aebef.el8 2024-01-16T04:34:38.598 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-449.ga70aebef.el8 2024-01-16T04:34:38.598 INFO:teuthology.task.install:The correct ceph version 16.2.14-449.ga70aebef is installed. 2024-01-16T04:34:38.599 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-16T04:34:38.600 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:38.600 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T04:34:38.665 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-16T04:34:38.666 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:38.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T04:34:38.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T04:34:38.801 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-16T04:34:38.801 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:38.801 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T04:34:38.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T04:34:38.934 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-16T04:34:38.935 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:38.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T04:34:39.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T04:34:39.069 INFO:teuthology.run_tasks:Running task ceph... 2024-01-16T04:34:39.125 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-16T04:34:39.125 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /var/log/ceph 2024-01-16T04:34:39.152 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-16T04:34:39.152 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T04:34:39.219 INFO:tasks.ceph:Creating extra log directories... 2024-01-16T04:34:39.219 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T04:34:39.295 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-16T04:34:39.296 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption_keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-16T04:34:39.296 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517318', 'branch': 'wip-yuri5-testing-2024-01-11-1300-pacific', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7517318', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2024-01-11-1300-pacific'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption_keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'}}, 'workunit': {'branch': 'wip-yuri5-testing-2024-01-11-1300-pacific', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9013, 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri5-testing-2024-01-11-1300-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'targets': {'smithi038.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMrCOmXvvC5QExnYAs37aTRoqTzxJQMTjdrxQAm9itOC+nKbW1cyOatQX/3ZlUqvyvCKNjuPlA+30FlZFkyfV9k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'rgw': {'client.0': None}}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-pacific', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-15_22:05:22', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-16T04:34:39.296 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T04:34:39.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T04:34:39.413 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:39.413 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T04:34:39.471 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T04:34:39.471 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-01-16T04:34:39.533 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T04:34:39.533 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T04:34:39.533 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 158616 Links: 1 2024-01-16T04:34:39.533 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T04:34:39.533 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T04:34:39.533 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-01-16 04:34:32.457362931 +0000 2024-01-16T04:34:39.533 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-01-16 04:30:27.040043047 +0000 2024-01-16T04:34:39.533 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-01-16 04:30:27.040043047 +0000 2024-01-16T04:34:39.533 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2024-01-16T04:34:39.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T04:34:39.598 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-01-16T04:34:39.598 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-01-16T04:34:39.599 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000275864 s, 1.9 MB/s 2024-01-16T04:34:39.599 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T04:34:39.659 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-01-16T04:34:39.717 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T04:34:39.717 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T04:34:39.717 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 158693 Links: 1 2024-01-16T04:34:39.717 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T04:34:39.718 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T04:34:39.718 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-01-16 04:34:32.457362931 +0000 2024-01-16T04:34:39.718 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-01-16 04:30:27.377036205 +0000 2024-01-16T04:34:39.718 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-01-16 04:30:27.377036205 +0000 2024-01-16T04:34:39.718 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2024-01-16T04:34:39.718 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T04:34:39.783 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-01-16T04:34:39.784 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-01-16T04:34:39.784 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000259045 s, 2.0 MB/s 2024-01-16T04:34:39.785 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T04:34:39.843 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-01-16T04:34:39.901 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T04:34:39.902 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T04:34:39.902 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 162187 Links: 1 2024-01-16T04:34:39.902 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T04:34:39.902 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T04:34:39.902 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-01-16 04:34:32.457362931 +0000 2024-01-16T04:34:39.902 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-01-16 04:30:27.730029025 +0000 2024-01-16T04:34:39.902 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-01-16 04:30:27.730029025 +0000 2024-01-16T04:34:39.902 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2024-01-16T04:34:39.902 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T04:34:39.967 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-01-16T04:34:39.967 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-01-16T04:34:39.967 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000255906 s, 2.0 MB/s 2024-01-16T04:34:39.969 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T04:34:40.029 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-01-16T04:34:40.087 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T04:34:40.087 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T04:34:40.087 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 163585 Links: 1 2024-01-16T04:34:40.087 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T04:34:40.087 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T04:34:40.087 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-01-16 04:34:32.457362931 +0000 2024-01-16T04:34:40.087 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-01-16 04:30:28.079021915 +0000 2024-01-16T04:34:40.087 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-01-16 04:30:28.079021915 +0000 2024-01-16T04:34:40.087 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2024-01-16T04:34:40.088 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T04:34:40.153 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-01-16T04:34:40.153 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-01-16T04:34:40.153 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000324381 s, 1.6 MB/s 2024-01-16T04:34:40.154 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T04:34:40.214 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-16T04:34:40.214 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-16T04:34:40.214 INFO:tasks.ceph:Generating config... 2024-01-16T04:34:40.215 INFO:tasks.ceph:[client] debug rgw = 20 2024-01-16T04:34:40.215 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-01-16T04:34:40.215 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-01-16T04:34:40.215 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption_keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-01-16T04:34:40.215 INFO:tasks.ceph:[client] setgroup = ceph 2024-01-16T04:34:40.215 INFO:tasks.ceph:[client] setuser = ceph 2024-01-16T04:34:40.216 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-16T04:34:40.216 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-16T04:34:40.216 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-16T04:34:40.216 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-16T04:34:40.216 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-16T04:34:40.216 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-16T04:34:40.216 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-16T04:34:40.216 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-16T04:34:40.216 INFO:tasks.ceph:Setting up mon.a... 2024-01-16T04:34:40.216 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-16T04:34:40.380 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/ceph.keyring 2024-01-16T04:34:40.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-16T04:34:40.424 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T04:34:40.492 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.38')] 2024-01-16T04:34:40.492 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.38'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption_keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2024-01-16T04:34:40.493 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:40.493 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T04:34:40.550 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.38 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:34:40.627 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:34:40.628 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:34:40.628 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:34:40.628 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: generated fsid 752b6117-c411-45ed-8b00-1e36318c8108 2024-01-16T04:34:40.628 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 0 2024-01-16T04:34:40.628 INFO:teuthology.orchestra.run.smithi038.stdout:fsid 752b6117-c411-45ed-8b00-1e36318c8108 2024-01-16T04:34:40.628 INFO:teuthology.orchestra.run.smithi038.stdout:last_changed 2024-01-16T04:34:40.632429+0000 2024-01-16T04:34:40.629 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-01-16T04:34:40.632429+0000 2024-01-16T04:34:40.629 INFO:teuthology.orchestra.run.smithi038.stdout:min_mon_release 0 (unknown) 2024-01-16T04:34:40.629 INFO:teuthology.orchestra.run.smithi038.stdout:election_strategy: 1 2024-01-16T04:34:40.629 INFO:teuthology.orchestra.run.smithi038.stdout:0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-01-16T04:34:40.629 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-01-16T04:34:40.630 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T04:34:40.687 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 752b6117-c411-45ed-8b00-1e36318c8108... 2024-01-16T04:34:40.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout: chdir = "" 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout: auth supported = cephx 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout: filestore xattr use omap = true 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout: mon clock drift allowed = 1.000 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout: osd crush chooseleaf type = 0 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout: auth debug = true 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.773 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on old message = true 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on bug = true 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: mon pg warn max object skew = 0 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default size = 2 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow primary affinity = true 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow pg remap = true 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on legacy crush tunables = false 2024-01-16T04:34:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on crush straw calc version zero = false 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on no sortbitwise = false 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on osd down out interval zero = false 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on too few osds = false 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: mon_allow_pool_size_one = true 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: osd default data pool replay window = 5 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: mon allow pool delete = true 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: mon cluster log file level = debug 2024-01-16T04:34:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: debug asserts on shutdown = true 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: mon health detail to clog = false 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: mon host = 172.21.15.38 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = 752b6117-c411-45ed-8b00-1e36318c8108 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout:[osd] 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: osd journal size = 100 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub load threshold = 5.0 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub max interval = 600 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: osd recover clone overlap = true 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: osd recovery max chunk = 1048576 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.776 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug shutdown = true 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug op order = true 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify stray on activate = true 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: osd open classes on start = true 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug pg log writeout = true 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: osd deep scrub update digest min age = 30 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: osd map max advance = 10 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: journal zero on create = true 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: filestore ondisk finisher threads = 3 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: filestore apply finisher threads = 3 2024-01-16T04:34:40.777 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: bdev debug aio = true 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug misdirected ops = true 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: debug osd = 20 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: osd shutdown pgref assert = True 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr] 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgr = 20 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: mgr/telemetry/nag = false 2024-01-16T04:34:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout:[mon] 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: debug paxos = 20 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: mon data avail warn = 5 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: mon mgr mkfs grace = 240 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd reporter subtree level = osd 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd prime pg temp = true 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T04:34:40.779 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout:[client] 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: rgw cache enabled = true 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable ops log = true 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable usage log = true 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: debug rgw = 20 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: rgw crypt require ssl = False 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: rgw crypt s3 kms backend = testing 2024-01-16T04:34:40.780 INFO:teuthology.orchestra.run.smithi038.stdout: rgw crypt s3 kms encryption_keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-01-16T04:34:40.781 INFO:teuthology.orchestra.run.smithi038.stdout: setgroup = ceph 2024-01-16T04:34:40.781 INFO:teuthology.orchestra.run.smithi038.stdout: setuser = ceph 2024-01-16T04:34:40.781 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.a] 2024-01-16T04:34:40.784 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-16T04:34:40.784 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-16T04:34:40.878 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-16T04:34:40.878 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:40.878 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-16T04:34:40.897 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:40.897 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-16T04:34:40.954 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi038.front.sepia.ceph.com 2024-01-16T04:34:40.954 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:40.954 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T04:34:40.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T04:34:41.032 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:41.032 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:34:41.091 INFO:tasks.ceph:Setting up mon nodes... 2024-01-16T04:34:41.091 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-16T04:34:41.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T04:34:41.181 INFO:teuthology.orchestra.run.smithi038.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T04:34:41.183 INFO:tasks.ceph:Setting up mds nodes... 2024-01-16T04:34:41.183 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-16T04:34:41.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-16T04:34:41.224 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-16T04:34:41.236 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-16T04:34:41.236 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-16T04:34:41.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-16T04:34:41.304 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-16T04:34:41.304 INFO:tasks.ceph:role: osd.0 2024-01-16T04:34:41.304 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com 2024-01-16T04:34:41.304 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T04:34:41.399 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T04:34:41.399 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T04:34:41.400 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T04:34:41.400 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T04:34:41.400 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T04:34:41.400 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-01-16T04:34:41.400 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T04:34:41.400 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T04:34:41.400 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T04:34:41.400 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T04:34:42.678 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-01-16T04:34:42.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-01-16T04:34:42.680 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-16T04:34:42.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-16T04:34:42.764 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-16T04:34:42.833 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-16T04:34:42.833 INFO:tasks.ceph:role: osd.1 2024-01-16T04:34:42.833 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com 2024-01-16T04:34:42.833 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T04:34:42.932 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T04:34:42.932 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T04:34:42.932 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T04:34:42.932 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T04:34:42.932 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T04:34:42.932 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-01-16T04:34:42.932 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T04:34:42.933 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T04:34:42.933 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T04:34:42.933 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T04:34:44.209 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-01-16T04:34:44.212 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-01-16T04:34:44.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-16T04:34:44.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-16T04:34:44.298 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-16T04:34:44.367 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-16T04:34:44.367 INFO:tasks.ceph:role: osd.2 2024-01-16T04:34:44.367 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com 2024-01-16T04:34:44.367 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T04:34:44.465 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T04:34:44.465 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T04:34:44.465 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T04:34:44.465 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T04:34:44.465 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T04:34:44.466 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-01-16T04:34:44.466 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T04:34:44.466 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T04:34:44.466 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T04:34:44.466 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T04:34:45.746 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-01-16T04:34:45.748 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-01-16T04:34:45.748 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-16T04:34:45.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-16T04:34:45.831 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:34:45.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:45.939+0000 7fe26e0f9380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-16T04:34:45.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:45.939+0000 7fe26e0f9380 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-16T04:34:45.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:45.939+0000 7fe26e0f9380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T04:34:45.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:45.939+0000 7fe26e0f9380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T04:34:45.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:45.939+0000 7fe26e0f9380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T04:34:45.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:45.940+0000 7fe26e0f9380 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-16T04:34:48.022 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-16T04:34:48.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:34:48.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:48.148+0000 7f2d2b2c2380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-16T04:34:48.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:48.148+0000 7f2d2b2c2380 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-16T04:34:48.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:48.148+0000 7f2d2b2c2380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T04:34:48.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:48.148+0000 7f2d2b2c2380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T04:34:48.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:48.148+0000 7f2d2b2c2380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T04:34:48.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:48.149+0000 7f2d2b2c2380 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-16T04:34:50.232 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-16T04:34:50.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:34:50.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:50.357+0000 7f526da49380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-16T04:34:50.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:50.357+0000 7f526da49380 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-16T04:34:50.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:50.357+0000 7f526da49380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T04:34:50.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:50.357+0000 7f526da49380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T04:34:50.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:50.357+0000 7f526da49380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T04:34:50.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:34:50.358+0000 7f526da49380 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-16T04:34:52.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-16T04:34:52.476 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-16T04:34:52.477 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:52.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-16T04:34:52.549 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:52.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-16T04:34:52.621 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:52.621 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-16T04:34:52.693 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:52.693 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-16T04:34:52.763 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:52.763 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-16T04:34:52.823 INFO:tasks.ceph:Adding keys to all mons... 2024-01-16T04:34:52.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T04:34:52.892 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.x] 2024-01-16T04:34:52.892 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBhB6Zlnx4TCxAAYwxO6xqNBkBCiuHJaJuLzQ== 2024-01-16T04:34:52.892 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.0] 2024-01-16T04:34:52.892 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBlB6ZlBX8IOBAA6O0VhYfXAMm9DHQVtzc/vg== 2024-01-16T04:34:52.892 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.1] 2024-01-16T04:34:52.892 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBoB6ZlWoXnCBAAb38g3w+B519wOtprMtk4rQ== 2024-01-16T04:34:52.892 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.2] 2024-01-16T04:34:52.892 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBqB6ZlZQ5pFRAAoZRUp/FoDBw+mEdIh1fwvQ== 2024-01-16T04:34:52.892 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2024-01-16T04:34:52.892 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBhB6ZlbhKcDRAAqz7nzo643HUuKMivIb4aUg== 2024-01-16T04:34:52.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T04:34:52.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T04:34:53.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T04:34:53.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T04:34:53.135 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T04:34:53.188 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-16T04:34:53.188 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-16T04:34:53.215 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T04:34:53.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-16T04:34:53.342 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:34:53.400 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-16T04:34:53.400 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-16T04:34:53.401 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-16T04:34:53.443 INFO:tasks.ceph.mon.a:Started 2024-01-16T04:34:53.443 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-16T04:34:53.443 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-16T04:34:53.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-16T04:34:53.445 INFO:tasks.ceph.mgr.x:Started 2024-01-16T04:34:53.445 INFO:tasks.ceph:Setting crush tunables to default 2024-01-16T04:34:53.445 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd crush tunables default 2024-01-16T04:34:53.661 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2024-01-16T04:34:53.671 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-16T04:34:53.672 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:53.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-16T04:34:53.699 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:53.699 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-16T04:34:53.766 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:34:53.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-16T04:34:53.833 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd new 4244da9f-308a-4ce5-9b56-a190f438a602 0 2024-01-16T04:34:53.859 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:53.862+0000 7f20d736b680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T04:34:53.932 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:53.936+0000 7f20d736b680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T04:34:54.067 INFO:teuthology.orchestra.run.smithi038.stdout:0 2024-01-16T04:34:54.079 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd new f6b91531-55fd-49cf-a96f-57dc23c4bd8a 1 2024-01-16T04:34:54.290 INFO:teuthology.orchestra.run.smithi038.stdout:1 2024-01-16T04:34:54.303 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd new ee72b7e9-cfcb-4142-ac3d-4d1e52bbb4a8 2 2024-01-16T04:34:54.387 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:54.390+0000 7f20d736b680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T04:34:54.524 INFO:teuthology.orchestra.run.smithi038.stdout:2 2024-01-16T04:34:54.537 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-16T04:34:54.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-16T04:34:54.539 INFO:tasks.ceph.osd.0:Started 2024-01-16T04:34:54.539 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-16T04:34:54.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-16T04:34:54.541 INFO:tasks.ceph.osd.1:Started 2024-01-16T04:34:54.542 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-16T04:34:54.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-16T04:34:54.544 INFO:tasks.ceph.osd.2:Started 2024-01-16T04:34:54.547 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T04:34:54.772 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:34:54.772 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":5,"fsid":"752b6117-c411-45ed-8b00-1e36318c8108","created":"2024-01-16T04:34:53.569874+0000","modified":"2024-01-16T04:34:54.517362+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T04:34:54.517362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"4244da9f-308a-4ce5-9b56-a190f438a602","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f6b91531-55fd-49cf-a96f-57dc23c4bd8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ee72b7e9-cfcb-4142-ac3d-4d1e52bbb4a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:34:54.785 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-16T04:34:54.785 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-16T04:34:55.086 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T04:34:55.161 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-01-16T04:34:55.164+0000 7fd674ccd380 -1 Falling back to public interface 2024-01-16T04:34:55.168 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-01-16T04:34:55.171+0000 7f40b8793380 -1 Falling back to public interface 2024-01-16T04:34:55.213 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-01-16T04:34:55.178+0000 7efd84486380 -1 Falling back to public interface 2024-01-16T04:34:55.214 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:55.180+0000 7f20d736b680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T04:34:55.250 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:55.253+0000 7f20d736b680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T04:34:55.313 INFO:teuthology.misc.health.smithi038.stdout: 2024-01-16T04:34:55.313 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":5,"fsid":"752b6117-c411-45ed-8b00-1e36318c8108","created":"2024-01-16T04:34:53.569874+0000","modified":"2024-01-16T04:34:54.517362+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T04:34:54.517362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"4244da9f-308a-4ce5-9b56-a190f438a602","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f6b91531-55fd-49cf-a96f-57dc23c4bd8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ee72b7e9-cfcb-4142-ac3d-4d1e52bbb4a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:34:55.325 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-01-16T04:34:55.446 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:55.449+0000 7f20d736b680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T04:34:55.900 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:55.903+0000 7f20d736b680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T04:34:56.006 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:56.010+0000 7f20d736b680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T04:34:56.165 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:56.168+0000 7f20d736b680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T04:34:56.244 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:56.248+0000 7f20d736b680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T04:34:56.316 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:56.320+0000 7f20d736b680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T04:34:56.587 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:56.590+0000 7f20d736b680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T04:34:56.792 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:56.795+0000 7f20d736b680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T04:34:57.486 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:57.489+0000 7f20d736b680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T04:34:57.557 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:57.561+0000 7f20d736b680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T04:34:57.640 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:57.643+0000 7f20d736b680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T04:34:57.794 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:57.797+0000 7f20d736b680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T04:34:57.868 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:57.872+0000 7f20d736b680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T04:34:57.888 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-01-16T04:34:57.892+0000 7f40b8793380 -1 osd.1 0 log_to_monitors {default=true} 2024-01-16T04:34:57.903 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-01-16T04:34:57.907+0000 7fd674ccd380 -1 osd.0 0 log_to_monitors {default=true} 2024-01-16T04:34:57.932 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-01-16T04:34:57.935+0000 7efd84486380 -1 osd.2 0 log_to_monitors {default=true} 2024-01-16T04:34:57.997 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:58.000+0000 7f20d736b680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T04:34:58.108 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:58.112+0000 7f20d736b680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T04:34:58.442 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:58.445+0000 7f20d736b680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T04:34:58.515 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:58.518+0000 7f20d736b680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T04:34:58.956 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-01-16T04:34:58.959+0000 7f40af134700 -1 osd.1 0 waiting for initial osdmap 2024-01-16T04:34:58.956 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-01-16T04:34:58.959+0000 7fd66b66e700 -1 osd.0 0 waiting for initial osdmap 2024-01-16T04:34:58.956 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-01-16T04:34:58.959+0000 7efd7ae27700 -1 osd.2 0 waiting for initial osdmap 2024-01-16T04:34:58.965 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-01-16T04:34:58.968+0000 7efd775a3700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T04:34:58.965 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-01-16T04:34:58.968+0000 7fd667dea700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T04:34:58.969 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-01-16T04:34:58.973+0000 7f40ab8b0700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T04:34:59.546 INFO:tasks.ceph.mgr.x.smithi038.stderr:2024-01-16T04:34:59.549+0000 7f208c863700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-01-16T04:35:01.628 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T04:35:01.933 INFO:teuthology.misc.health.smithi038.stdout: 2024-01-16T04:35:01.933 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":9,"fsid":"752b6117-c411-45ed-8b00-1e36318c8108","created":"2024-01-16T04:34:53.569874+0000","modified":"2024-01-16T04:35:00.969270+0000","last_up_change":"2024-01-16T04:34:59.963811+0000","last_in_change":"2024-01-16T04:34:54.517362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-16T04:35:00.023083+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4244da9f-308a-4ce5-9b56-a190f438a602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":40407},{"type":"v1","addr":"172.21.15.38:6801","nonce":40407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":40407},{"type":"v1","addr":"172.21.15.38:6803","nonce":40407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":40407},{"type":"v1","addr":"172.21.15.38:6807","nonce":40407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":40407},{"type":"v1","addr":"172.21.15.38:6805","nonce":40407}]},"public_addr":"172.21.15.38:6801/40407","cluster_addr":"172.21.15.38:6803/40407","heartbeat_back_addr":"172.21.15.38:6807/40407","heartbeat_front_addr":"172.21.15.38:6805/40407","state":["exists","up"]},{"osd":1,"uuid":"f6b91531-55fd-49cf-a96f-57dc23c4bd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":40408},{"type":"v1","addr":"172.21.15.38:6809","nonce":40408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":40408},{"type":"v1","addr":"172.21.15.38:6811","nonce":40408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":40408},{"type":"v1","addr":"172.21.15.38:6815","nonce":40408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":40408},{"type":"v1","addr":"172.21.15.38:6813","nonce":40408}]},"public_addr":"172.21.15.38:6809/40408","cluster_addr":"172.21.15.38:6811/40408","heartbeat_back_addr":"172.21.15.38:6815/40408","heartbeat_front_addr":"172.21.15.38:6813/40408","state":["exists","up"]},{"osd":2,"uuid":"ee72b7e9-cfcb-4142-ac3d-4d1e52bbb4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":40437},{"type":"v1","addr":"172.21.15.38:6817","nonce":40437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":40437},{"type":"v1","addr":"172.21.15.38:6819","nonce":40437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":40437},{"type":"v1","addr":"172.21.15.38:6823","nonce":40437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":40437},{"type":"v1","addr":"172.21.15.38:6821","nonce":40437}]},"public_addr":"172.21.15.38:6817/40437","cluster_addr":"172.21.15.38:6819/40437","heartbeat_back_addr":"172.21.15.38:6823/40437","heartbeat_front_addr":"172.21.15.38:6821/40437","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:34:58.931224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:34:58.897675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:34:58.911909+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:35:01.946 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-01-16T04:35:01.946 INFO:tasks.ceph:Creating RBD pool 2024-01-16T04:35:01.946 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-16T04:35:02.983 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'rbd' created 2024-01-16T04:35:02.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-16T04:35:03.983 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-16T04:35:03.998 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-16T04:35:03.999 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-16T04:35:03.999 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-16T04:35:03.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-16T04:35:03.999 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-16T04:35:04.352 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:04.364 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":39992},{"type":"v1","addr":"172.21.15.38:6825","nonce":39992}]},"active_addr":"172.21.15.38:6825/39992","active_change":"2024-01-16T04:34:58.533136+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":3490504923}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":2591325884}]}]} 2024-01-16T04:35:04.367 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-16T04:35:04.367 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-16T04:35:04.367 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T04:35:04.666 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:04.666 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":12,"fsid":"752b6117-c411-45ed-8b00-1e36318c8108","created":"2024-01-16T04:34:53.569874+0000","modified":"2024-01-16T04:35:03.978956+0000","last_up_change":"2024-01-16T04:34:59.963811+0000","last_in_change":"2024-01-16T04:34:54.517362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-16T04:35:00.023083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-16T04:35:02.294714+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4244da9f-308a-4ce5-9b56-a190f438a602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":40407},{"type":"v1","addr":"172.21.15.38:6801","nonce":40407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":40407},{"type":"v1","addr":"172.21.15.38:6803","nonce":40407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":40407},{"type":"v1","addr":"172.21.15.38:6807","nonce":40407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":40407},{"type":"v1","addr":"172.21.15.38:6805","nonce":40407}]},"public_addr":"172.21.15.38:6801/40407","cluster_addr":"172.21.15.38:6803/40407","heartbeat_back_addr":"172.21.15.38:6807/40407","heartbeat_front_addr":"172.21.15.38:6805/40407","state":["exists","up"]},{"osd":1,"uuid":"f6b91531-55fd-49cf-a96f-57dc23c4bd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":40408},{"type":"v1","addr":"172.21.15.38:6809","nonce":40408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":40408},{"type":"v1","addr":"172.21.15.38:6811","nonce":40408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":40408},{"type":"v1","addr":"172.21.15.38:6815","nonce":40408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":40408},{"type":"v1","addr":"172.21.15.38:6813","nonce":40408}]},"public_addr":"172.21.15.38:6809/40408","cluster_addr":"172.21.15.38:6811/40408","heartbeat_back_addr":"172.21.15.38:6815/40408","heartbeat_front_addr":"172.21.15.38:6813/40408","state":["exists","up"]},{"osd":2,"uuid":"ee72b7e9-cfcb-4142-ac3d-4d1e52bbb4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":40437},{"type":"v1","addr":"172.21.15.38:6817","nonce":40437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":40437},{"type":"v1","addr":"172.21.15.38:6819","nonce":40437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":40437},{"type":"v1","addr":"172.21.15.38:6823","nonce":40437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":40437},{"type":"v1","addr":"172.21.15.38:6821","nonce":40437}]},"public_addr":"172.21.15.38:6817/40437","cluster_addr":"172.21.15.38:6819/40437","heartbeat_back_addr":"172.21.15.38:6823/40437","heartbeat_front_addr":"172.21.15.38:6821/40437","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:34:58.931224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:34:58.897675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:34:58.911909+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:35:04.680 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-16T04:35:04.680 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T04:35:04.983 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:04.983 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":13,"fsid":"752b6117-c411-45ed-8b00-1e36318c8108","created":"2024-01-16T04:34:53.569874+0000","modified":"2024-01-16T04:35:04.982037+0000","last_up_change":"2024-01-16T04:34:59.963811+0000","last_in_change":"2024-01-16T04:34:54.517362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-16T04:35:00.023083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-16T04:35:02.294714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4244da9f-308a-4ce5-9b56-a190f438a602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":40407},{"type":"v1","addr":"172.21.15.38:6801","nonce":40407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":40407},{"type":"v1","addr":"172.21.15.38:6803","nonce":40407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":40407},{"type":"v1","addr":"172.21.15.38:6807","nonce":40407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":40407},{"type":"v1","addr":"172.21.15.38:6805","nonce":40407}]},"public_addr":"172.21.15.38:6801/40407","cluster_addr":"172.21.15.38:6803/40407","heartbeat_back_addr":"172.21.15.38:6807/40407","heartbeat_front_addr":"172.21.15.38:6805/40407","state":["exists","up"]},{"osd":1,"uuid":"f6b91531-55fd-49cf-a96f-57dc23c4bd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":40408},{"type":"v1","addr":"172.21.15.38:6809","nonce":40408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":40408},{"type":"v1","addr":"172.21.15.38:6811","nonce":40408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":40408},{"type":"v1","addr":"172.21.15.38:6815","nonce":40408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":40408},{"type":"v1","addr":"172.21.15.38:6813","nonce":40408}]},"public_addr":"172.21.15.38:6809/40408","cluster_addr":"172.21.15.38:6811/40408","heartbeat_back_addr":"172.21.15.38:6815/40408","heartbeat_front_addr":"172.21.15.38:6813/40408","state":["exists","up"]},{"osd":2,"uuid":"ee72b7e9-cfcb-4142-ac3d-4d1e52bbb4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":40437},{"type":"v1","addr":"172.21.15.38:6817","nonce":40437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":40437},{"type":"v1","addr":"172.21.15.38:6819","nonce":40437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":40437},{"type":"v1","addr":"172.21.15.38:6823","nonce":40437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":40437},{"type":"v1","addr":"172.21.15.38:6821","nonce":40437}]},"public_addr":"172.21.15.38:6817/40437","cluster_addr":"172.21.15.38:6819/40437","heartbeat_back_addr":"172.21.15.38:6823/40437","heartbeat_front_addr":"172.21.15.38:6821/40437","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:34:58.931224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:34:58.897675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:34:58.911909+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:35:04.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-16T04:35:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:34359738370 2024-01-16T04:35:05.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-16T04:35:05.299 INFO:teuthology.orchestra.run.smithi038.stdout:34359738370 2024-01-16T04:35:05.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-16T04:35:05.450 INFO:teuthology.orchestra.run.smithi038.stdout:34359738370 2024-01-16T04:35:05.450 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T04:35:05.749 INFO:teuthology.orchestra.run.smithi038.stdout:34359738369 2024-01-16T04:35:05.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2024-01-16T04:35:06.763 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T04:35:07.077 INFO:teuthology.orchestra.run.smithi038.stdout:34359738371 2024-01-16T04:35:07.090 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738371 for osd.0 2024-01-16T04:35:07.090 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T04:35:07.388 INFO:teuthology.orchestra.run.smithi038.stdout:34359738371 2024-01-16T04:35:07.401 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738371 for osd.1 2024-01-16T04:35:07.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T04:35:07.700 INFO:teuthology.orchestra.run.smithi038.stdout:34359738371 2024-01-16T04:35:07.714 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738371 for osd.2 2024-01-16T04:35:07.714 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T04:35:07.714 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T04:35:08.009 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:08.011 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-01-16T04:35:08.022 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-01-16T04:35:06.542663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":891600,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":313681200,"statfs":{"total":322122547200,"available":321209548800,"internally_reserved":0,"allocated":319488,"data_stored":86889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.003870+0000","last_change":"2024-01-16T04:35:04.000348+0000","last_active":"2024-01-16T04:35:04.003870+0000","last_peered":"2024-01-16T04:35:04.003870+0000","last_clean":"2024-01-16T04:35:04.003870+0000","last_became_active":"2024-01-16T04:35:04.000075+0000","last_became_peered":"2024-01-16T04:35:04.000075+0000","last_unstale":"2024-01-16T04:35:04.003870+0000","last_undegraded":"2024-01-16T04:35:04.003870+0000","last_fullsized":"2024-01-16T04:35:04.003870+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.003874+0000","last_change":"2024-01-16T04:35:04.000383+0000","last_active":"2024-01-16T04:35:04.003874+0000","last_peered":"2024-01-16T04:35:04.003874+0000","last_clean":"2024-01-16T04:35:04.003874+0000","last_became_active":"2024-01-16T04:35:04.000102+0000","last_became_peered":"2024-01-16T04:35:04.000102+0000","last_unstale":"2024-01-16T04:35:04.003874+0000","last_undegraded":"2024-01-16T04:35:04.003874+0000","last_fullsized":"2024-01-16T04:35:04.003874+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.005628+0000","last_change":"2024-01-16T04:35:04.001138+0000","last_active":"2024-01-16T04:35:04.005628+0000","last_peered":"2024-01-16T04:35:04.005628+0000","last_clean":"2024-01-16T04:35:04.005628+0000","last_became_active":"2024-01-16T04:35:04.000767+0000","last_became_peered":"2024-01-16T04:35:04.000767+0000","last_unstale":"2024-01-16T04:35:04.005628+0000","last_undegraded":"2024-01-16T04:35:04.005628+0000","last_fullsized":"2024-01-16T04:35:04.005628+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.000632+0000","last_change":"2024-01-16T04:35:04.000632+0000","last_active":"2024-01-16T04:35:04.000632+0000","last_peered":"2024-01-16T04:35:04.000632+0000","last_clean":"2024-01-16T04:35:04.000632+0000","last_became_active":"2024-01-16T04:35:04.000340+0000","last_became_peered":"2024-01-16T04:35:04.000340+0000","last_unstale":"2024-01-16T04:35:04.000632+0000","last_undegraded":"2024-01-16T04:35:04.000632+0000","last_fullsized":"2024-01-16T04:35:04.000632+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:03.990973+0000","last_change":"2024-01-16T04:35:01.992221+0000","last_active":"2024-01-16T04:35:03.990973+0000","last_peered":"2024-01-16T04:35:03.990973+0000","last_clean":"2024-01-16T04:35:03.990973+0000","last_became_active":"2024-01-16T04:35:01.991947+0000","last_became_peered":"2024-01-16T04:35:01.991947+0000","last_unstale":"2024-01-16T04:35:03.990973+0000","last_undegraded":"2024-01-16T04:35:03.990973+0000","last_fullsized":"2024-01-16T04:35:03.990973+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:00.969270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:00.969270+0000","last_clean_scrub_stamp":"2024-01-16T04:35:00.969270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.003887+0000","last_change":"2024-01-16T04:35:04.000303+0000","last_active":"2024-01-16T04:35:04.003887+0000","last_peered":"2024-01-16T04:35:04.003887+0000","last_clean":"2024-01-16T04:35:04.003887+0000","last_became_active":"2024-01-16T04:35:03.999379+0000","last_became_peered":"2024-01-16T04:35:03.999379+0000","last_unstale":"2024-01-16T04:35:04.003887+0000","last_undegraded":"2024-01-16T04:35:04.003887+0000","last_fullsized":"2024-01-16T04:35:04.003887+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.001009+0000","last_change":"2024-01-16T04:35:04.001009+0000","last_active":"2024-01-16T04:35:04.001009+0000","last_peered":"2024-01-16T04:35:04.001009+0000","last_clean":"2024-01-16T04:35:04.001009+0000","last_became_active":"2024-01-16T04:35:04.000724+0000","last_became_peered":"2024-01-16T04:35:04.000724+0000","last_unstale":"2024-01-16T04:35:04.001009+0000","last_undegraded":"2024-01-16T04:35:04.001009+0000","last_fullsized":"2024-01-16T04:35:04.001009+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.000494+0000","last_change":"2024-01-16T04:35:04.000494+0000","last_active":"2024-01-16T04:35:04.000494+0000","last_peered":"2024-01-16T04:35:04.000494+0000","last_clean":"2024-01-16T04:35:04.000494+0000","last_became_active":"2024-01-16T04:35:04.000198+0000","last_became_peered":"2024-01-16T04:35:04.000198+0000","last_unstale":"2024-01-16T04:35:04.000494+0000","last_undegraded":"2024-01-16T04:35:04.000494+0000","last_fullsized":"2024-01-16T04:35:04.000494+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.005640+0000","last_change":"2024-01-16T04:35:04.001305+0000","last_active":"2024-01-16T04:35:04.005640+0000","last_peered":"2024-01-16T04:35:04.005640+0000","last_clean":"2024-01-16T04:35:04.005640+0000","last_became_active":"2024-01-16T04:35:04.001001+0000","last_became_peered":"2024-01-16T04:35:04.001001+0000","last_unstale":"2024-01-16T04:35:04.005640+0000","last_undegraded":"2024-01-16T04:35:04.005640+0000","last_fullsized":"2024-01-16T04:35:04.005640+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T04:35:08.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T04:35:08.319 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:08.321 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-01-16T04:35:08.332 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-01-16T04:35:06.542663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":891600,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":313681200,"statfs":{"total":322122547200,"available":321209548800,"internally_reserved":0,"allocated":319488,"data_stored":86889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.003870+0000","last_change":"2024-01-16T04:35:04.000348+0000","last_active":"2024-01-16T04:35:04.003870+0000","last_peered":"2024-01-16T04:35:04.003870+0000","last_clean":"2024-01-16T04:35:04.003870+0000","last_became_active":"2024-01-16T04:35:04.000075+0000","last_became_peered":"2024-01-16T04:35:04.000075+0000","last_unstale":"2024-01-16T04:35:04.003870+0000","last_undegraded":"2024-01-16T04:35:04.003870+0000","last_fullsized":"2024-01-16T04:35:04.003870+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.003874+0000","last_change":"2024-01-16T04:35:04.000383+0000","last_active":"2024-01-16T04:35:04.003874+0000","last_peered":"2024-01-16T04:35:04.003874+0000","last_clean":"2024-01-16T04:35:04.003874+0000","last_became_active":"2024-01-16T04:35:04.000102+0000","last_became_peered":"2024-01-16T04:35:04.000102+0000","last_unstale":"2024-01-16T04:35:04.003874+0000","last_undegraded":"2024-01-16T04:35:04.003874+0000","last_fullsized":"2024-01-16T04:35:04.003874+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.005628+0000","last_change":"2024-01-16T04:35:04.001138+0000","last_active":"2024-01-16T04:35:04.005628+0000","last_peered":"2024-01-16T04:35:04.005628+0000","last_clean":"2024-01-16T04:35:04.005628+0000","last_became_active":"2024-01-16T04:35:04.000767+0000","last_became_peered":"2024-01-16T04:35:04.000767+0000","last_unstale":"2024-01-16T04:35:04.005628+0000","last_undegraded":"2024-01-16T04:35:04.005628+0000","last_fullsized":"2024-01-16T04:35:04.005628+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.000632+0000","last_change":"2024-01-16T04:35:04.000632+0000","last_active":"2024-01-16T04:35:04.000632+0000","last_peered":"2024-01-16T04:35:04.000632+0000","last_clean":"2024-01-16T04:35:04.000632+0000","last_became_active":"2024-01-16T04:35:04.000340+0000","last_became_peered":"2024-01-16T04:35:04.000340+0000","last_unstale":"2024-01-16T04:35:04.000632+0000","last_undegraded":"2024-01-16T04:35:04.000632+0000","last_fullsized":"2024-01-16T04:35:04.000632+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:03.990973+0000","last_change":"2024-01-16T04:35:01.992221+0000","last_active":"2024-01-16T04:35:03.990973+0000","last_peered":"2024-01-16T04:35:03.990973+0000","last_clean":"2024-01-16T04:35:03.990973+0000","last_became_active":"2024-01-16T04:35:01.991947+0000","last_became_peered":"2024-01-16T04:35:01.991947+0000","last_unstale":"2024-01-16T04:35:03.990973+0000","last_undegraded":"2024-01-16T04:35:03.990973+0000","last_fullsized":"2024-01-16T04:35:03.990973+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:00.969270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:00.969270+0000","last_clean_scrub_stamp":"2024-01-16T04:35:00.969270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.003887+0000","last_change":"2024-01-16T04:35:04.000303+0000","last_active":"2024-01-16T04:35:04.003887+0000","last_peered":"2024-01-16T04:35:04.003887+0000","last_clean":"2024-01-16T04:35:04.003887+0000","last_became_active":"2024-01-16T04:35:03.999379+0000","last_became_peered":"2024-01-16T04:35:03.999379+0000","last_unstale":"2024-01-16T04:35:04.003887+0000","last_undegraded":"2024-01-16T04:35:04.003887+0000","last_fullsized":"2024-01-16T04:35:04.003887+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.001009+0000","last_change":"2024-01-16T04:35:04.001009+0000","last_active":"2024-01-16T04:35:04.001009+0000","last_peered":"2024-01-16T04:35:04.001009+0000","last_clean":"2024-01-16T04:35:04.001009+0000","last_became_active":"2024-01-16T04:35:04.000724+0000","last_became_peered":"2024-01-16T04:35:04.000724+0000","last_unstale":"2024-01-16T04:35:04.001009+0000","last_undegraded":"2024-01-16T04:35:04.001009+0000","last_fullsized":"2024-01-16T04:35:04.001009+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.000494+0000","last_change":"2024-01-16T04:35:04.000494+0000","last_active":"2024-01-16T04:35:04.000494+0000","last_peered":"2024-01-16T04:35:04.000494+0000","last_clean":"2024-01-16T04:35:04.000494+0000","last_became_active":"2024-01-16T04:35:04.000198+0000","last_became_peered":"2024-01-16T04:35:04.000198+0000","last_unstale":"2024-01-16T04:35:04.000494+0000","last_undegraded":"2024-01-16T04:35:04.000494+0000","last_fullsized":"2024-01-16T04:35:04.000494+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:35:04.005640+0000","last_change":"2024-01-16T04:35:04.001305+0000","last_active":"2024-01-16T04:35:04.005640+0000","last_peered":"2024-01-16T04:35:04.005640+0000","last_clean":"2024-01-16T04:35:04.005640+0000","last_became_active":"2024-01-16T04:35:04.001001+0000","last_became_peered":"2024-01-16T04:35:04.001001+0000","last_unstale":"2024-01-16T04:35:04.005640+0000","last_undegraded":"2024-01-16T04:35:04.005640+0000","last_fullsized":"2024-01-16T04:35:04.005640+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:35:02.975601+0000","last_clean_scrub_stamp":"2024-01-16T04:35:02.975601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T04:35:08.333 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-16T04:35:08.333 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-16T04:35:08.333 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-16T04:35:08.333 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-16T04:35:08.669 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:08.682 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-16T04:35:08.683 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-16T04:35:08.683 INFO:teuthology.run_tasks:Running task rgw... 2024-01-16T04:35:08.697 DEBUG:tasks.rgw:config is {'client.0': None} 2024-01-16T04:35:08.697 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2024-01-16T04:35:08.697 INFO:tasks.rgw:Creating data pools 2024-01-16T04:35:08.697 DEBUG:tasks.rgw:Obtaining remote for client client.0 2024-01-16T04:35:08.698 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2024-01-16T04:35:09.550 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'default.rgw.buckets.data' created 2024-01-16T04:35:09.568 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2024-01-16T04:35:10.577 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-01-16T04:35:10.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2024-01-16T04:35:11.577 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'default.rgw.buckets.index' created 2024-01-16T04:35:11.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2024-01-16T04:35:12.593 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-01-16T04:35:12.612 DEBUG:tasks.rgw:Pools created 2024-01-16T04:35:12.613 INFO:tasks.rgw:Starting rgw... 2024-01-16T04:35:12.613 INFO:tasks.rgw:rgw client.0 config is {} 2024-01-16T04:35:12.613 INFO:tasks.rgw:Using civetweb as radosgw frontend 2024-01-16T04:35:12.613 INFO:tasks.rgw.client.0:Restarting daemon 2024-01-16T04:35:12.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2024-01-16T04:35:12.615 INFO:tasks.rgw.client.0:Started 2024-01-16T04:35:12.616 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi038.front.sepia.ceph.com:80/ 2024-01-16T04:35:12.616 DEBUG:teuthology.orchestra.run.smithi038:> curl http://smithi038.front.sepia.ceph.com:80/ 2024-01-16T04:35:12.653 INFO:teuthology.orchestra.run.smithi038.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:35:12.653 INFO:teuthology.orchestra.run.smithi038.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:35:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-01-16T04:35:12.655 INFO:teuthology.orchestra.run.smithi038.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi038.front.sepia.ceph.com port 80: Connection refused 2024-01-16T04:35:13.656 DEBUG:teuthology.orchestra.run.smithi038:> curl http://smithi038.front.sepia.ceph.com:80/ 2024-01-16T04:35:13.682 INFO:teuthology.orchestra.run.smithi038.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:35:13.682 INFO:teuthology.orchestra.run.smithi038.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:35:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-01-16T04:35:13.684 INFO:teuthology.orchestra.run.smithi038.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi038.front.sepia.ceph.com port 80: Connection refused 2024-01-16T04:35:15.685 DEBUG:teuthology.orchestra.run.smithi038:> curl http://smithi038.front.sepia.ceph.com:80/ 2024-01-16T04:35:15.711 INFO:teuthology.orchestra.run.smithi038.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:35:15.712 INFO:teuthology.orchestra.run.smithi038.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:35:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-01-16T04:35:15.713 INFO:teuthology.orchestra.run.smithi038.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi038.front.sepia.ceph.com port 80: Connection refused 2024-01-16T04:35:19.715 DEBUG:teuthology.orchestra.run.smithi038:> curl http://smithi038.front.sepia.ceph.com:80/ 2024-01-16T04:35:19.742 INFO:teuthology.orchestra.run.smithi038.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:35:19.742 INFO:teuthology.orchestra.run.smithi038.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:35:19.744 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-01-16T04:35:19.744 INFO:teuthology.orchestra.run.smithi038.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi038.front.sepia.ceph.com port 80: Connection refused 2024-01-16T04:35:21.687 INFO:tasks.rgw.client.0.smithi038.stdout:2024-01-16T04:35:21.690+0000 7ff62853d880 -1 IMPORTANT: the civetweb frontend is now deprecated and will be removed in a future release 2024-01-16T04:35:27.745 DEBUG:teuthology.orchestra.run.smithi038:> curl http://smithi038.front.sepia.ceph.com:80/ 2024-01-16T04:35:27.771 INFO:teuthology.orchestra.run.smithi038.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:35:27.771 INFO:teuthology.orchestra.run.smithi038.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:35:27.772 INFO:teuthology.orchestra.run.smithi038.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-01-16T04:35:27.773 INFO:teuthology.orchestra.run.smithi038.stdout:anonymous 2024-01-16T04:35:27.774 INFO:teuthology.run_tasks:Running task s3tests... 2024-01-16T04:35:27.787 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-pacific', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}} 2024-01-16T04:35:27.787 INFO:tasks.s3tests:Downloading s3-tests... 2024-01-16T04:35:27.788 INFO:tasks.s3tests:Using branch 'ceph-pacific' for s3tests 2024-01-16T04:35:27.788 DEBUG:teuthology.orchestra.run.smithi038:> git clone -b ceph-pacific https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests 2024-01-16T04:35:27.835 INFO:teuthology.orchestra.run.smithi038.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests'... 2024-01-16T04:35:28.508 INFO:tasks.s3tests:Creating rgw users... 2024-01-16T04:35:28.508 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2024-01-16T04:35:28.508 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key TVYAECPWOXPTEUUVKKVE --secret UDoWzWVLp4Ace75P4Z2XO0QqR3JufFfXZZp6FsffTE+KEFg58D0W4w== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2024-01-16T04:35:28.601 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:35:28.601 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:35:28.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.611+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.612+0000 7fdbdf0c97c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.612+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.612+0000 7fdbdf0c97c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.612+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.613+0000 7fdbdf0c97c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.613+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.613+0000 7fdbdf0c97c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:28.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.620+0000 7fdbdf0c97c0 20 RGWRados::pool_iterate: got zone_info.e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 2024-01-16T04:35:28.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.620+0000 7fdbdf0c97c0 20 RGWRados::pool_iterate: got zonegroup_info.78faec5d-171d-49c3-930a-c335c902808d 2024-01-16T04:35:28.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.620+0000 7fdbdf0c97c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:35:28.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.620+0000 7fdbdf0c97c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:35:28.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.620+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.621+0000 7fdbdf0c97c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:28.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.621+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.621+0000 7fdbdf0c97c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:35:28.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.621+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.621+0000 7fdbdf0c97c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:28.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.621+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.621+0000 7fdbdf0c97c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:35:28.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.621+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 20 zonegroup default 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 10 Cannot find current period zone using local zone 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.622+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.623+0000 7fdbdf0c97c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.623+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.623+0000 7fdbdf0c97c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:28.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.623+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.623+0000 7fdbdf0c97c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:35:28.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.623+0000 7fdbdf0c97c0 20 zone default found 2024-01-16T04:35:28.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.623+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.623+0000 7fdbdf0c97c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.623+0000 7fdbdf0c97c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.624+0000 7fdbdf0c97c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.624+0000 7fdbdf0c97c0 20 started sync module instance, tier type = 2024-01-16T04:35:28.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.624+0000 7fdbdf0c97c0 20 started zone id=e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 (name=default) with tier type = 2024-01-16T04:35:28.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.633+0000 7fdbdf0c97c0 20 add_watcher() i=0 2024-01-16T04:35:28.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.633+0000 7fdbdf0c97c0 20 add_watcher() i=1 2024-01-16T04:35:28.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.633+0000 7fdbdf0c97c0 20 add_watcher() i=2 2024-01-16T04:35:28.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.633+0000 7fdbdf0c97c0 20 add_watcher() i=3 2024-01-16T04:35:28.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.633+0000 7fdbdf0c97c0 20 add_watcher() i=4 2024-01-16T04:35:28.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.633+0000 7fdbdf0c97c0 20 add_watcher() i=5 2024-01-16T04:35:28.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.633+0000 7fdbdf0c97c0 20 add_watcher() i=6 2024-01-16T04:35:28.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.634+0000 7fdbdf0c97c0 20 add_watcher() i=7 2024-01-16T04:35:28.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.634+0000 7fdbdf0c97c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:35:28.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.635+0000 7fdb9bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:35:28.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.635+0000 7fdbdf0c97c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:35:28.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.635+0000 7fdbdf0c97c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:35:28.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.639+0000 7fdbdf0c97c0 5 note: GC not initialized 2024-01-16T04:35:28.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.640+0000 7fdb0affd700 20 reqs_thread_entry: start 2024-01-16T04:35:28.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.640+0000 7fdb08ff9700 20 reqs_thread_entry: start 2024-01-16T04:35:28.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdbdf0c97c0 20 init_complete bucket index max shards: 11 2024-01-16T04:35:28.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdbdf0c97c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:35:28.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdafeffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:35:58 2024 2024-01-16T04:35:28.638 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:35:28.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdafe7fc700 20 reqs_thread_entry: start 2024-01-16T04:35:28.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdbdf0c97c0 20 get_system_obj_state: rctx=0x7ffc59d71b88 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55678f5dad50 s->prefetch_data=0 2024-01-16T04:35:28.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdbdf0c97c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-01-16T04:35:28.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdbdf0c97c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2024-01-16T04:35:28.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdbdf0c97c0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:35:28.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdbdf0c97c0 20 get_system_obj_state: rctx=0x7ffc59d71b98 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x55678f5dad50 s->prefetch_data=0 2024-01-16T04:35:28.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdbdf0c97c0 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2024-01-16T04:35:28.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdbdf0c97c0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2024-01-16T04:35:28.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdbdf0c97c0 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2024-01-16T04:35:28.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdbdf0c97c0 20 get_system_obj_state: rctx=0x7ffc59d71b98 obj=default.rgw.meta:users.keys:TVYAECPWOXPTEUUVKKVE state=0x55678f5dad50 s->prefetch_data=0 2024-01-16T04:35:28.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.641+0000 7fdbdf0c97c0 10 cache get: name=default.rgw.meta+users.keys+TVYAECPWOXPTEUUVKKVE : miss 2024-01-16T04:35:28.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.642+0000 7fdbdf0c97c0 10 cache put: name=default.rgw.meta+users.keys+TVYAECPWOXPTEUUVKKVE info.flags=0x0 2024-01-16T04:35:28.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.642+0000 7fdbdf0c97c0 10 adding default.rgw.meta+users.keys+TVYAECPWOXPTEUUVKKVE to cache LRU end 2024-01-16T04:35:28.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.642+0000 7fdbdf0c97c0 20 get_system_obj_state: rctx=0x7ffc59d714b8 obj=default.rgw.meta:users.keys:TVYAECPWOXPTEUUVKKVE state=0x55678f5dad50 s->prefetch_data=0 2024-01-16T04:35:28.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.642+0000 7fdbdf0c97c0 10 cache get: name=default.rgw.meta+users.keys+TVYAECPWOXPTEUUVKKVE : hit (negative entry) 2024-01-16T04:35:28.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.642+0000 7fdbdf0c97c0 20 get_system_obj_state: rctx=0x7ffc59d717f8 obj=default.rgw.meta:users.keys:TVYAECPWOXPTEUUVKKVE state=0x55678f5dad50 s->prefetch_data=0 2024-01-16T04:35:28.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.642+0000 7fdbdf0c97c0 10 cache get: name=default.rgw.meta+users.keys+TVYAECPWOXPTEUUVKKVE : hit (negative entry) 2024-01-16T04:35:28.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.643+0000 7fdbdf0c97c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-01-16T04:35:28.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.643+0000 7fdbdf0c97c0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:35:28.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.643+0000 7fdbdf0c97c0 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-01-16T04:35:28.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.643+0000 7fdbbbfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149696 cookie 93903276618416 notifier 4199 bl.length()=640 2024-01-16T04:35:28.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.643+0000 7fdbbbfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-01-16T04:35:28.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.643+0000 7fdbbbfff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:35:28.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.645+0000 7fdbdf0c97c0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2024-01-16T04:35:28.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.645+0000 7fdbdf0c97c0 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2024-01-16T04:35:28.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.645+0000 7fdbdf0c97c0 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2024-01-16T04:35:28.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.645+0000 7fdbbbfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149696 cookie 93903276640496 notifier 4199 bl.length()=194 2024-01-16T04:35:28.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.645+0000 7fdbbbfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2024-01-16T04:35:28.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.645+0000 7fdbbbfff700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2024-01-16T04:35:28.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.647+0000 7fdbdf0c97c0 10 cache put: name=default.rgw.meta+users.keys+TVYAECPWOXPTEUUVKKVE info.flags=0x7 2024-01-16T04:35:28.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.647+0000 7fdbdf0c97c0 10 moving default.rgw.meta+users.keys+TVYAECPWOXPTEUUVKKVE to cache LRU end 2024-01-16T04:35:28.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.647+0000 7fdbdf0c97c0 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:TVYAECPWOXPTEUUVKKVE, ofs0, ns] 2024-01-16T04:35:28.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.648+0000 7fdbbb7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149697 cookie 93903276626560 notifier 4199 bl.length()=186 2024-01-16T04:35:28.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.648+0000 7fdbbb7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+TVYAECPWOXPTEUUVKKVE info.flags=0x7 2024-01-16T04:35:28.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.648+0000 7fdbbb7fe700 10 rgw watcher librados: moving default.rgw.meta+users.keys+TVYAECPWOXPTEUUVKKVE to cache LRU end 2024-01-16T04:35:28.645 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-01-16T04:35:28.645 INFO:teuthology.orchestra.run.smithi038.stdout: "user_id": "foo.client.0", 2024-01-16T04:35:28.645 INFO:teuthology.orchestra.run.smithi038.stdout: "display_name": "Mr. foo.client.0", 2024-01-16T04:35:28.645 INFO:teuthology.orchestra.run.smithi038.stdout: "email": "foo.client.0+test@test.test", 2024-01-16T04:35:28.645 INFO:teuthology.orchestra.run.smithi038.stdout: "suspended": 0, 2024-01-16T04:35:28.645 INFO:teuthology.orchestra.run.smithi038.stdout: "max_buckets": 1000, 2024-01-16T04:35:28.645 INFO:teuthology.orchestra.run.smithi038.stdout: "subusers": [], 2024-01-16T04:35:28.645 INFO:teuthology.orchestra.run.smithi038.stdout: "keys": [ 2024-01-16T04:35:28.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: "user": "foo.client.0", 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "TVYAECPWOXPTEUUVKKVE", 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "UDoWzWVLp4Ace75P4Z2XO0QqR3JufFfXZZp6FsffTE+KEFg58D0W4w==" 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: "swift_keys": [], 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": [ 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "user-policy", 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: "perm": "*" 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: "op_mask": "read, write, delete", 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: "default_placement": "", 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: "default_storage_class": "", 2024-01-16T04:35:28.646 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_tags": [], 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_quota": { 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "user_quota": { 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "temp_url_keys": [], 2024-01-16T04:35:28.647 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "rgw", 2024-01-16T04:35:28.648 INFO:teuthology.orchestra.run.smithi038.stdout: "mfa_ids": [] 2024-01-16T04:35:28.648 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-01-16T04:35:28.648 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:28.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.650+0000 7fdbdf0c97c0 20 remove_watcher() i=0 2024-01-16T04:35:28.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.650+0000 7fdbdf0c97c0 2 removed watcher, disabling cache 2024-01-16T04:35:28.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.651+0000 7fdbdf0c97c0 20 remove_watcher() i=1 2024-01-16T04:35:28.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.652+0000 7fdbdf0c97c0 20 remove_watcher() i=2 2024-01-16T04:35:28.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.653+0000 7fdbdf0c97c0 20 remove_watcher() i=3 2024-01-16T04:35:28.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.654+0000 7fdbdf0c97c0 20 remove_watcher() i=4 2024-01-16T04:35:28.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.655+0000 7fdbdf0c97c0 20 remove_watcher() i=5 2024-01-16T04:35:28.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.656+0000 7fdbdf0c97c0 20 remove_watcher() i=6 2024-01-16T04:35:28.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.657+0000 7fdbdf0c97c0 20 remove_watcher() i=7 2024-01-16T04:35:28.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.657+0000 7fdb9bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:35:28.659 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 0677277422 --totp-seed F7HZI7BNUAL27J44HCVYGJT6VSCHA36NCBBUYAH4LSN6IRVTTNPZF4UQMD436B2X --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-01-16T04:35:28.708 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:35:28.708 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:35:28.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.718+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.719+0000 7fa8e4aa17c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.719+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.719+0000 7fa8e4aa17c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.719+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.719+0000 7fa8e4aa17c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.719+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.720+0000 7fa8e4aa17c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:28.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.727+0000 7fa8e4aa17c0 20 RGWRados::pool_iterate: got zone_info.e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 2024-01-16T04:35:28.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.727+0000 7fa8e4aa17c0 20 RGWRados::pool_iterate: got zonegroup_info.78faec5d-171d-49c3-930a-c335c902808d 2024-01-16T04:35:28.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.727+0000 7fa8e4aa17c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:35:28.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.727+0000 7fa8e4aa17c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:35:28.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.727+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.728+0000 7fa8e4aa17c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:28.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.728+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.728+0000 7fa8e4aa17c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:35:28.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.728+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.728+0000 7fa8e4aa17c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:28.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.728+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.728+0000 7fa8e4aa17c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.728+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.729+0000 7fa8e4aa17c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.729+0000 7fa8e4aa17c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.729+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.729+0000 7fa8e4aa17c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.729+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.729+0000 7fa8e4aa17c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.729+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.729+0000 7fa8e4aa17c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.729+0000 7fa8e4aa17c0 20 zonegroup default 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.729+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.730+0000 7fa8e4aa17c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.730+0000 7fa8e4aa17c0 10 Cannot find current period zone using local zone 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.730+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.730+0000 7fa8e4aa17c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.730+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.730+0000 7fa8e4aa17c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.730+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.730+0000 7fa8e4aa17c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.730+0000 7fa8e4aa17c0 20 zone default found 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.730+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.730+0000 7fa8e4aa17c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.730+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.731+0000 7fa8e4aa17c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.731+0000 7fa8e4aa17c0 20 started sync module instance, tier type = 2024-01-16T04:35:28.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.731+0000 7fa8e4aa17c0 20 started zone id=e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 (name=default) with tier type = 2024-01-16T04:35:28.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.745+0000 7fa8e4aa17c0 20 add_watcher() i=0 2024-01-16T04:35:28.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.745+0000 7fa8e4aa17c0 20 add_watcher() i=1 2024-01-16T04:35:28.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.745+0000 7fa8e4aa17c0 20 add_watcher() i=2 2024-01-16T04:35:28.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.745+0000 7fa8e4aa17c0 20 add_watcher() i=3 2024-01-16T04:35:28.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.745+0000 7fa8e4aa17c0 20 add_watcher() i=4 2024-01-16T04:35:28.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.745+0000 7fa8e4aa17c0 20 add_watcher() i=5 2024-01-16T04:35:28.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.745+0000 7fa8e4aa17c0 20 add_watcher() i=6 2024-01-16T04:35:28.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.746+0000 7fa8e4aa17c0 20 add_watcher() i=7 2024-01-16T04:35:28.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.746+0000 7fa8e4aa17c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:35:28.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.747+0000 7fa8a17e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:35:28.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.747+0000 7fa8e4aa17c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:35:28.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.747+0000 7fa8e4aa17c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:35:28.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.751+0000 7fa8e4aa17c0 5 note: GC not initialized 2024-01-16T04:35:28.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.752+0000 7fa80ffff700 20 reqs_thread_entry: start 2024-01-16T04:35:28.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.752+0000 7fa80dffb700 20 reqs_thread_entry: start 2024-01-16T04:35:28.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.752+0000 7fa8e4aa17c0 20 init_complete bucket index max shards: 11 2024-01-16T04:35:28.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.753+0000 7fa8e4aa17c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:35:28.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.753+0000 7fa7fbfff700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:35:59 2024 2024-01-16T04:35:28.750 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:35:28.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.753+0000 7fa7fb7fe700 20 reqs_thread_entry: start 2024-01-16T04:35:28.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.753+0000 7fa8e4aa17c0 20 get_system_obj_state: rctx=0x7ffc81054ca8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55d036ac9d50 s->prefetch_data=0 2024-01-16T04:35:28.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.753+0000 7fa8e4aa17c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-01-16T04:35:28.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.753+0000 7fa8e4aa17c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2024-01-16T04:35:28.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.753+0000 7fa8e4aa17c0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:35:28.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.753+0000 7fa8e4aa17c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:35:28.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.753+0000 7fa8e4aa17c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:35:28.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.753+0000 7fa8e4aa17c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:35:28.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.753+0000 7fa8e4aa17c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:28.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.754+0000 7fa8e4aa17c0 20 rados_obj.operate() r=0 bl.length=455 2024-01-16T04:35:28.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.754+0000 7fa8e4aa17c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x11 2024-01-16T04:35:28.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:28.754+0000 7fa8e4aa17c0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:35:30.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.580+0000 7fa8e4aa17c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-01-16T04:35:30.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.580+0000 7fa8e4aa17c0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:35:30.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.580+0000 7fa8e4aa17c0 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-01-16T04:35:30.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.580+0000 7fa8cd3fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084290 cookie 94352760155440 notifier 4205 bl.length()=654 2024-01-16T04:35:30.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.580+0000 7fa8cd3fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-01-16T04:35:30.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.580+0000 7fa8cd3fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:35:30.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.583+0000 7fa8e4aa17c0 20 remove_watcher() i=0 2024-01-16T04:35:30.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.583+0000 7fa8e4aa17c0 2 removed watcher, disabling cache 2024-01-16T04:35:30.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.584+0000 7fa8e4aa17c0 20 remove_watcher() i=1 2024-01-16T04:35:30.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.585+0000 7fa8e4aa17c0 20 remove_watcher() i=2 2024-01-16T04:35:30.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.586+0000 7fa8e4aa17c0 20 remove_watcher() i=3 2024-01-16T04:35:30.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.587+0000 7fa8e4aa17c0 20 remove_watcher() i=4 2024-01-16T04:35:30.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.589+0000 7fa8e4aa17c0 20 remove_watcher() i=5 2024-01-16T04:35:30.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.590+0000 7fa8e4aa17c0 20 remove_watcher() i=6 2024-01-16T04:35:30.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.594+0000 7fa8e4aa17c0 20 remove_watcher() i=7 2024-01-16T04:35:30.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.595+0000 7fa8a17e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:35:30.598 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2024-01-16T04:35:30.598 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key EUTPDQQLUHNECISCHSYU --secret v9J7JXK7AbYsdlaKxqy0aBSpupYOG7iwRRUetcoc2qXfUOixUgiN5g== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2024-01-16T04:35:30.652 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:35:30.652 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:35:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.662+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.663+0000 7fbb4e0b47c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.663+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.663+0000 7fbb4e0b47c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.663+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.663+0000 7fbb4e0b47c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.663+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.663+0000 7fbb4e0b47c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.671+0000 7fbb4e0b47c0 20 RGWRados::pool_iterate: got zone_info.e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 2024-01-16T04:35:30.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.671+0000 7fbb4e0b47c0 20 RGWRados::pool_iterate: got zonegroup_info.78faec5d-171d-49c3-930a-c335c902808d 2024-01-16T04:35:30.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.671+0000 7fbb4e0b47c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:35:30.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.671+0000 7fbb4e0b47c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:35:30.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.671+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.672+0000 7fbb4e0b47c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.672+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.672+0000 7fbb4e0b47c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:35:30.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.672+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.672+0000 7fbb4e0b47c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.672+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.673+0000 7fbb4e0b47c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.673+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.673+0000 7fbb4e0b47c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.673+0000 7fbb4e0b47c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.673+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.673+0000 7fbb4e0b47c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.673+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.673+0000 7fbb4e0b47c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.673+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.673+0000 7fbb4e0b47c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.673+0000 7fbb4e0b47c0 20 zonegroup default 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.673+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.674+0000 7fbb4e0b47c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.674+0000 7fbb4e0b47c0 10 Cannot find current period zone using local zone 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.674+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.674+0000 7fbb4e0b47c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.674+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.674+0000 7fbb4e0b47c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.674+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.674+0000 7fbb4e0b47c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.674+0000 7fbb4e0b47c0 20 zone default found 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.674+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.675+0000 7fbb4e0b47c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.675+0000 7fbb4e0b47c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.675+0000 7fbb4e0b47c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.675+0000 7fbb4e0b47c0 20 started sync module instance, tier type = 2024-01-16T04:35:30.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.675+0000 7fbb4e0b47c0 20 started zone id=e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 (name=default) with tier type = 2024-01-16T04:35:30.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.685+0000 7fbb4e0b47c0 20 add_watcher() i=0 2024-01-16T04:35:30.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.685+0000 7fbb4e0b47c0 20 add_watcher() i=1 2024-01-16T04:35:30.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.685+0000 7fbb4e0b47c0 20 add_watcher() i=2 2024-01-16T04:35:30.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.685+0000 7fbb4e0b47c0 20 add_watcher() i=3 2024-01-16T04:35:30.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.685+0000 7fbb4e0b47c0 20 add_watcher() i=4 2024-01-16T04:35:30.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.685+0000 7fbb4e0b47c0 20 add_watcher() i=5 2024-01-16T04:35:30.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.685+0000 7fbb4e0b47c0 20 add_watcher() i=6 2024-01-16T04:35:30.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.687+0000 7fbb4e0b47c0 20 add_watcher() i=7 2024-01-16T04:35:30.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.687+0000 7fbb4e0b47c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:35:30.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.688+0000 7fbb0a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:35:30.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.688+0000 7fbb4e0b47c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:35:30.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.688+0000 7fbb4e0b47c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:35:30.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.692+0000 7fbb4e0b47c0 5 note: GC not initialized 2024-01-16T04:35:30.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.693+0000 7fba797fa700 20 reqs_thread_entry: start 2024-01-16T04:35:30.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.694+0000 7fba72ffd700 20 reqs_thread_entry: start 2024-01-16T04:35:30.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.694+0000 7fbb4e0b47c0 20 init_complete bucket index max shards: 11 2024-01-16T04:35:30.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.694+0000 7fbb4e0b47c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:35:30.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.694+0000 7fba717fa700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:36:01 2024 2024-01-16T04:35:30.692 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:35:30.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.694+0000 7fba70ff9700 20 reqs_thread_entry: start 2024-01-16T04:35:30.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.694+0000 7fbb4e0b47c0 20 get_system_obj_state: rctx=0x7fff843a8508 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55b8d19abd50 s->prefetch_data=0 2024-01-16T04:35:30.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.694+0000 7fbb4e0b47c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-01-16T04:35:30.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.694+0000 7fbb4e0b47c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2024-01-16T04:35:30.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.694+0000 7fbb4e0b47c0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:35:30.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.694+0000 7fbb4e0b47c0 20 get_system_obj_state: rctx=0x7fff843a8518 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x55b8d19abd50 s->prefetch_data=0 2024-01-16T04:35:30.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.694+0000 7fbb4e0b47c0 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2024-01-16T04:35:30.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.695+0000 7fbb4e0b47c0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2024-01-16T04:35:30.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.695+0000 7fbb4e0b47c0 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2024-01-16T04:35:30.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.695+0000 7fbb4e0b47c0 20 get_system_obj_state: rctx=0x7fff843a8518 obj=default.rgw.meta:users.keys:EUTPDQQLUHNECISCHSYU state=0x55b8d19abd50 s->prefetch_data=0 2024-01-16T04:35:30.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.695+0000 7fbb4e0b47c0 10 cache get: name=default.rgw.meta+users.keys+EUTPDQQLUHNECISCHSYU : miss 2024-01-16T04:35:30.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.695+0000 7fbb4e0b47c0 10 cache put: name=default.rgw.meta+users.keys+EUTPDQQLUHNECISCHSYU info.flags=0x0 2024-01-16T04:35:30.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.695+0000 7fbb4e0b47c0 10 adding default.rgw.meta+users.keys+EUTPDQQLUHNECISCHSYU to cache LRU end 2024-01-16T04:35:30.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.695+0000 7fbb4e0b47c0 20 get_system_obj_state: rctx=0x7fff843a7e38 obj=default.rgw.meta:users.keys:EUTPDQQLUHNECISCHSYU state=0x55b8d19abd50 s->prefetch_data=0 2024-01-16T04:35:30.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.695+0000 7fbb4e0b47c0 10 cache get: name=default.rgw.meta+users.keys+EUTPDQQLUHNECISCHSYU : hit (negative entry) 2024-01-16T04:35:30.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.695+0000 7fbb4e0b47c0 20 get_system_obj_state: rctx=0x7fff843a8178 obj=default.rgw.meta:users.keys:EUTPDQQLUHNECISCHSYU state=0x55b8d19abd50 s->prefetch_data=0 2024-01-16T04:35:30.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.695+0000 7fbb4e0b47c0 10 cache get: name=default.rgw.meta+users.keys+EUTPDQQLUHNECISCHSYU : hit (negative entry) 2024-01-16T04:35:30.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.696+0000 7fbb4e0b47c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-01-16T04:35:30.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.696+0000 7fbb4e0b47c0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:35:30.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.696+0000 7fbb4e0b47c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-01-16T04:35:30.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.697+0000 7fbb2affd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084289 cookie 94252280242352 notifier 4211 bl.length()=640 2024-01-16T04:35:30.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.697+0000 7fbb2affd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-01-16T04:35:30.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.697+0000 7fbb2affd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:35:30.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.699+0000 7fbb4e0b47c0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2024-01-16T04:35:30.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.699+0000 7fbb4e0b47c0 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2024-01-16T04:35:30.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.699+0000 7fbb4e0b47c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2024-01-16T04:35:30.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.699+0000 7fbb2affd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084290 cookie 94252280242352 notifier 4211 bl.length()=194 2024-01-16T04:35:30.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.699+0000 7fbb2affd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2024-01-16T04:35:30.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.699+0000 7fbb2affd700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2024-01-16T04:35:30.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.700+0000 7fbb4e0b47c0 10 cache put: name=default.rgw.meta+users.keys+EUTPDQQLUHNECISCHSYU info.flags=0x7 2024-01-16T04:35:30.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.700+0000 7fbb4e0b47c0 10 moving default.rgw.meta+users.keys+EUTPDQQLUHNECISCHSYU to cache LRU end 2024-01-16T04:35:30.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.700+0000 7fbb4e0b47c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:EUTPDQQLUHNECISCHSYU, ofs0, ns] 2024-01-16T04:35:30.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.701+0000 7fbb2a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084291 cookie 94252280242352 notifier 4211 bl.length()=186 2024-01-16T04:35:30.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.701+0000 7fbb2a7fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+EUTPDQQLUHNECISCHSYU info.flags=0x7 2024-01-16T04:35:30.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.701+0000 7fbb2a7fc700 10 rgw watcher librados: moving default.rgw.meta+users.keys+EUTPDQQLUHNECISCHSYU to cache LRU end 2024-01-16T04:35:30.698 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-01-16T04:35:30.698 INFO:teuthology.orchestra.run.smithi038.stdout: "user_id": "bar.client.0", 2024-01-16T04:35:30.698 INFO:teuthology.orchestra.run.smithi038.stdout: "display_name": "Mr. bar.client.0", 2024-01-16T04:35:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: "email": "bar.client.0+test@test.test", 2024-01-16T04:35:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: "suspended": 0, 2024-01-16T04:35:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: "max_buckets": 1000, 2024-01-16T04:35:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: "subusers": [], 2024-01-16T04:35:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: "keys": [ 2024-01-16T04:35:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-01-16T04:35:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: "user": "bar.client.0", 2024-01-16T04:35:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "EUTPDQQLUHNECISCHSYU", 2024-01-16T04:35:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "v9J7JXK7AbYsdlaKxqy0aBSpupYOG7iwRRUetcoc2qXfUOixUgiN5g==" 2024-01-16T04:35:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-01-16T04:35:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-01-16T04:35:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: "swift_keys": [], 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": [ 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "user-policy", 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: "perm": "*" 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: "op_mask": "read, write, delete", 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: "default_placement": "", 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: "default_storage_class": "", 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_tags": [], 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_quota": { 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2024-01-16T04:35:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: "user_quota": { 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: "temp_url_keys": [], 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "rgw", 2024-01-16T04:35:30.701 INFO:teuthology.orchestra.run.smithi038.stdout: "mfa_ids": [] 2024-01-16T04:35:30.702 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-01-16T04:35:30.702 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:30.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.703+0000 7fbb4e0b47c0 20 remove_watcher() i=0 2024-01-16T04:35:30.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.703+0000 7fbb4e0b47c0 2 removed watcher, disabling cache 2024-01-16T04:35:30.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.704+0000 7fbb4e0b47c0 20 remove_watcher() i=1 2024-01-16T04:35:30.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.705+0000 7fbb4e0b47c0 20 remove_watcher() i=2 2024-01-16T04:35:30.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.706+0000 7fbb4e0b47c0 20 remove_watcher() i=3 2024-01-16T04:35:30.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.707+0000 7fbb4e0b47c0 20 remove_watcher() i=4 2024-01-16T04:35:30.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.708+0000 7fbb4e0b47c0 20 remove_watcher() i=5 2024-01-16T04:35:30.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.709+0000 7fbb4e0b47c0 20 remove_watcher() i=6 2024-01-16T04:35:30.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.710+0000 7fbb4e0b47c0 20 remove_watcher() i=7 2024-01-16T04:35:30.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.710+0000 7fbb0a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:35:30.711 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 6863634906 --totp-seed VCTLAEUC6TFA4EEKURDOUK5HOPRZFGAKRDY2UVQQICK72MNBLWFGWIJ3TDXKBK5R --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-01-16T04:35:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:35:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:35:30.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.770+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.771+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.771+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.771+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.771+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.772+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.772+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.772+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.779+0000 7fbb8ff6d7c0 20 RGWRados::pool_iterate: got zone_info.e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 2024-01-16T04:35:30.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.779+0000 7fbb8ff6d7c0 20 RGWRados::pool_iterate: got zonegroup_info.78faec5d-171d-49c3-930a-c335c902808d 2024-01-16T04:35:30.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.779+0000 7fbb8ff6d7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:35:30.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.779+0000 7fbb8ff6d7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:35:30.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.779+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.779+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.779+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.780+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:35:30.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.780+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.780+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.780+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.780+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:35:30.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.780+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.780+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.780+0000 7fbb8ff6d7c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:35:30.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.780+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.780+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.780+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.781+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.781+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.781+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.781+0000 7fbb8ff6d7c0 20 zonegroup default 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.781+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.781+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.781+0000 7fbb8ff6d7c0 10 Cannot find current period zone using local zone 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.781+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.781+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.781+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.781+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.781+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.782+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:35:30.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.782+0000 7fbb8ff6d7c0 20 zone default found 2024-01-16T04:35:30.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.782+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.782+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.782+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.782+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.782+0000 7fbb8ff6d7c0 20 started sync module instance, tier type = 2024-01-16T04:35:30.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.782+0000 7fbb8ff6d7c0 20 started zone id=e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 (name=default) with tier type = 2024-01-16T04:35:30.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.792+0000 7fbb8ff6d7c0 20 add_watcher() i=0 2024-01-16T04:35:30.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.792+0000 7fbb8ff6d7c0 20 add_watcher() i=1 2024-01-16T04:35:30.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.792+0000 7fbb8ff6d7c0 20 add_watcher() i=2 2024-01-16T04:35:30.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.792+0000 7fbb8ff6d7c0 20 add_watcher() i=3 2024-01-16T04:35:30.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.792+0000 7fbb8ff6d7c0 20 add_watcher() i=4 2024-01-16T04:35:30.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.792+0000 7fbb8ff6d7c0 20 add_watcher() i=5 2024-01-16T04:35:30.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.792+0000 7fbb8ff6d7c0 20 add_watcher() i=6 2024-01-16T04:35:30.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.793+0000 7fbb8ff6d7c0 20 add_watcher() i=7 2024-01-16T04:35:30.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.793+0000 7fbb8ff6d7c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:35:30.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.794+0000 7fbac7fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:35:30.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.794+0000 7fbb8ff6d7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:35:30.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.794+0000 7fbb8ff6d7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:35:30.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.798+0000 7fbb8ff6d7c0 5 note: GC not initialized 2024-01-16T04:35:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.799+0000 7fbab6ffd700 20 reqs_thread_entry: start 2024-01-16T04:35:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.799+0000 7fbab57fa700 20 reqs_thread_entry: start 2024-01-16T04:35:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.799+0000 7fbb8ff6d7c0 20 init_complete bucket index max shards: 11 2024-01-16T04:35:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.799+0000 7fbb8ff6d7c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:35:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.799+0000 7fbaaeffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:36:01 2024 2024-01-16T04:35:30.797 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:35:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.799+0000 7fbaae7fc700 20 reqs_thread_entry: start 2024-01-16T04:35:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.800+0000 7fbb8ff6d7c0 20 get_system_obj_state: rctx=0x7ffddcbd9218 obj=default.rgw.meta:users.uid:bar.client.0 state=0x563ee059ed50 s->prefetch_data=0 2024-01-16T04:35:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.800+0000 7fbb8ff6d7c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-01-16T04:35:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.800+0000 7fbb8ff6d7c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2024-01-16T04:35:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.800+0000 7fbb8ff6d7c0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:35:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.800+0000 7fbb8ff6d7c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:35:30.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.800+0000 7fbb8ff6d7c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:35:30.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.800+0000 7fbb8ff6d7c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:35:30.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.800+0000 7fbb8ff6d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.801+0000 7fbb8ff6d7c0 20 rados_obj.operate() r=0 bl.length=455 2024-01-16T04:35:30.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.801+0000 7fbb8ff6d7c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x11 2024-01-16T04:35:30.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.801+0000 7fbb8ff6d7c0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:35:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.803+0000 7fbb8ff6d7c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-01-16T04:35:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.803+0000 7fbb8ff6d7c0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:35:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.803+0000 7fbb8ff6d7c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-01-16T04:35:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.803+0000 7fbb790c9700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084292 cookie 94828053265664 notifier 4217 bl.length()=654 2024-01-16T04:35:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.803+0000 7fbb790c9700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-01-16T04:35:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.803+0000 7fbb790c9700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:35:30.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.806+0000 7fbb8ff6d7c0 20 remove_watcher() i=0 2024-01-16T04:35:30.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.806+0000 7fbb8ff6d7c0 2 removed watcher, disabling cache 2024-01-16T04:35:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.806+0000 7fbb8ff6d7c0 20 remove_watcher() i=1 2024-01-16T04:35:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.808+0000 7fbb8ff6d7c0 20 remove_watcher() i=2 2024-01-16T04:35:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.809+0000 7fbb8ff6d7c0 20 remove_watcher() i=3 2024-01-16T04:35:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.809+0000 7fbb8ff6d7c0 20 remove_watcher() i=4 2024-01-16T04:35:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.810+0000 7fbb8ff6d7c0 20 remove_watcher() i=5 2024-01-16T04:35:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.811+0000 7fbb8ff6d7c0 20 remove_watcher() i=6 2024-01-16T04:35:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.812+0000 7fbb8ff6d7c0 20 remove_watcher() i=7 2024-01-16T04:35:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.813+0000 7fbac7fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:35:30.814 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2024-01-16T04:35:30.814 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key LMHZFGXTIMPAKJZGYPFN --secret sXa449XQBq2gaehWhnimqN4yrXKDCJbzvqLKh4YW8wnw5F5FC35sDw== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2024-01-16T04:35:30.864 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:35:30.864 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:35:30.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.874+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.874+0000 7feee4b917c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.874+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.875+0000 7feee4b917c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.875+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.875+0000 7feee4b917c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.875+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.875+0000 7feee4b917c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.882+0000 7feee4b917c0 20 RGWRados::pool_iterate: got zone_info.e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 2024-01-16T04:35:30.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.882+0000 7feee4b917c0 20 RGWRados::pool_iterate: got zonegroup_info.78faec5d-171d-49c3-930a-c335c902808d 2024-01-16T04:35:30.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.882+0000 7feee4b917c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:35:30.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.882+0000 7feee4b917c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:35:30.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.882+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.883+0000 7feee4b917c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.883+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.883+0000 7feee4b917c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:35:30.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.883+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.883+0000 7feee4b917c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.883+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.883+0000 7feee4b917c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:35:30.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.883+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.884+0000 7feee4b917c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.884+0000 7feee4b917c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:35:30.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.884+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.884+0000 7feee4b917c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.884+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.884+0000 7feee4b917c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.884+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.884+0000 7feee4b917c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.884+0000 7feee4b917c0 20 zonegroup default 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.884+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.885+0000 7feee4b917c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.885+0000 7feee4b917c0 10 Cannot find current period zone using local zone 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.885+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.885+0000 7feee4b917c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.885+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.885+0000 7feee4b917c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.885+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.885+0000 7feee4b917c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:35:30.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.885+0000 7feee4b917c0 20 zone default found 2024-01-16T04:35:30.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.885+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.886+0000 7feee4b917c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.886+0000 7feee4b917c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.886+0000 7feee4b917c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.886+0000 7feee4b917c0 20 started sync module instance, tier type = 2024-01-16T04:35:30.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.886+0000 7feee4b917c0 20 started zone id=e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 (name=default) with tier type = 2024-01-16T04:35:30.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.897+0000 7feee4b917c0 20 add_watcher() i=0 2024-01-16T04:35:30.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.897+0000 7feee4b917c0 20 add_watcher() i=1 2024-01-16T04:35:30.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.897+0000 7feee4b917c0 20 add_watcher() i=2 2024-01-16T04:35:30.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.897+0000 7feee4b917c0 20 add_watcher() i=3 2024-01-16T04:35:30.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.897+0000 7feee4b917c0 20 add_watcher() i=4 2024-01-16T04:35:30.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.897+0000 7feee4b917c0 20 add_watcher() i=5 2024-01-16T04:35:30.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.897+0000 7feee4b917c0 20 add_watcher() i=6 2024-01-16T04:35:30.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.898+0000 7feee4b917c0 20 add_watcher() i=7 2024-01-16T04:35:30.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.898+0000 7feee4b917c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:35:30.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.899+0000 7feea17e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:35:30.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.899+0000 7feee4b917c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:35:30.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.899+0000 7feee4b917c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:35:30.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.903+0000 7feee4b917c0 5 note: GC not initialized 2024-01-16T04:35:30.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.904+0000 7fee0ffff700 20 reqs_thread_entry: start 2024-01-16T04:35:30.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.904+0000 7fee0dffb700 20 reqs_thread_entry: start 2024-01-16T04:35:30.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.904+0000 7feee4b917c0 20 init_complete bucket index max shards: 11 2024-01-16T04:35:30.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.904+0000 7feee4b917c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:35:30.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.904+0000 7fedfbfff700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:36:01 2024 2024-01-16T04:35:30.902 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.904+0000 7fedfb7fe700 20 reqs_thread_entry: start 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.904+0000 7feee4b917c0 20 get_system_obj_state: rctx=0x7ffd8f1c7f28 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55c5e7fabd50 s->prefetch_data=0 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.904+0000 7feee4b917c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 20 get_system_obj_state: rctx=0x7ffd8f1c7f38 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x55c5e7fabd50 s->prefetch_data=0 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 20 get_system_obj_state: rctx=0x7ffd8f1c7f38 obj=default.rgw.meta:users.keys:LMHZFGXTIMPAKJZGYPFN state=0x55c5e7fabd50 s->prefetch_data=0 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 10 cache get: name=default.rgw.meta+users.keys+LMHZFGXTIMPAKJZGYPFN : miss 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 10 cache put: name=default.rgw.meta+users.keys+LMHZFGXTIMPAKJZGYPFN info.flags=0x0 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 10 adding default.rgw.meta+users.keys+LMHZFGXTIMPAKJZGYPFN to cache LRU end 2024-01-16T04:35:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 20 get_system_obj_state: rctx=0x7ffd8f1c7858 obj=default.rgw.meta:users.keys:LMHZFGXTIMPAKJZGYPFN state=0x55c5e7fabd50 s->prefetch_data=0 2024-01-16T04:35:30.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 10 cache get: name=default.rgw.meta+users.keys+LMHZFGXTIMPAKJZGYPFN : hit (negative entry) 2024-01-16T04:35:30.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 20 get_system_obj_state: rctx=0x7ffd8f1c7b98 obj=default.rgw.meta:users.keys:LMHZFGXTIMPAKJZGYPFN state=0x55c5e7fabd50 s->prefetch_data=0 2024-01-16T04:35:30.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.905+0000 7feee4b917c0 10 cache get: name=default.rgw.meta+users.keys+LMHZFGXTIMPAKJZGYPFN : hit (negative entry) 2024-01-16T04:35:30.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.906+0000 7feee4b917c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-01-16T04:35:30.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.906+0000 7feee4b917c0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:35:30.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.906+0000 7feee4b917c0 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-01-16T04:35:30.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.907+0000 7feecdced700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084291 cookie 94308490222752 notifier 4223 bl.length()=714 2024-01-16T04:35:30.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.907+0000 7feecdced700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-01-16T04:35:30.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.907+0000 7feecdced700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:35:30.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.908+0000 7feee4b917c0 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2024-01-16T04:35:30.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.908+0000 7feee4b917c0 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2024-01-16T04:35:30.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.908+0000 7feee4b917c0 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2024-01-16T04:35:30.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.909+0000 7feecdced700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084292 cookie 94308490212976 notifier 4223 bl.length()=224 2024-01-16T04:35:30.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.909+0000 7feecdced700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2024-01-16T04:35:30.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.909+0000 7feecdced700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2024-01-16T04:35:30.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.910+0000 7feee4b917c0 10 cache put: name=default.rgw.meta+users.keys+LMHZFGXTIMPAKJZGYPFN info.flags=0x7 2024-01-16T04:35:30.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.910+0000 7feee4b917c0 10 moving default.rgw.meta+users.keys+LMHZFGXTIMPAKJZGYPFN to cache LRU end 2024-01-16T04:35:30.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.910+0000 7feee4b917c0 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:LMHZFGXTIMPAKJZGYPFN, ofs0, ns] 2024-01-16T04:35:30.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.910+0000 7feecd4ec700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084288 cookie 94308490209824 notifier 4223 bl.length()=201 2024-01-16T04:35:30.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.910+0000 7feecd4ec700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+LMHZFGXTIMPAKJZGYPFN info.flags=0x7 2024-01-16T04:35:30.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.910+0000 7feecd4ec700 10 rgw watcher librados: moving default.rgw.meta+users.keys+LMHZFGXTIMPAKJZGYPFN to cache LRU end 2024-01-16T04:35:30.908 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-01-16T04:35:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "user_id": "testx$tenanteduser.client.0", 2024-01-16T04:35:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2024-01-16T04:35:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2024-01-16T04:35:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "suspended": 0, 2024-01-16T04:35:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "max_buckets": 1000, 2024-01-16T04:35:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "subusers": [], 2024-01-16T04:35:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "keys": [ 2024-01-16T04:35:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-01-16T04:35:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "user": "testx$tenanteduser.client.0", 2024-01-16T04:35:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "LMHZFGXTIMPAKJZGYPFN", 2024-01-16T04:35:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "sXa449XQBq2gaehWhnimqN4yrXKDCJbzvqLKh4YW8wnw5F5FC35sDw==" 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "swift_keys": [], 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": [ 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "user-policy", 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "perm": "*" 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "op_mask": "read, write, delete", 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "default_placement": "", 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "default_storage_class": "", 2024-01-16T04:35:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_tags": [], 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_quota": { 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "user_quota": { 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-01-16T04:35:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "temp_url_keys": [], 2024-01-16T04:35:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "rgw", 2024-01-16T04:35:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mfa_ids": [] 2024-01-16T04:35:30.911 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-01-16T04:35:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:30.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.912+0000 7feee4b917c0 20 remove_watcher() i=0 2024-01-16T04:35:30.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.912+0000 7feee4b917c0 2 removed watcher, disabling cache 2024-01-16T04:35:30.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.913+0000 7feee4b917c0 20 remove_watcher() i=1 2024-01-16T04:35:30.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.914+0000 7feee4b917c0 20 remove_watcher() i=2 2024-01-16T04:35:30.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.915+0000 7feee4b917c0 20 remove_watcher() i=3 2024-01-16T04:35:30.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.916+0000 7feee4b917c0 20 remove_watcher() i=4 2024-01-16T04:35:30.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.917+0000 7feee4b917c0 20 remove_watcher() i=5 2024-01-16T04:35:30.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.918+0000 7feee4b917c0 20 remove_watcher() i=6 2024-01-16T04:35:30.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.919+0000 7feee4b917c0 20 remove_watcher() i=7 2024-01-16T04:35:30.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.919+0000 7feea17e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:35:30.921 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 7888840671 --totp-seed TP27GKAXXGN2CU6ROFARYKAEGZMA3Y32FH6T6BQMUAJ35TWUCYCWYSUIZCMOIYDT --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-01-16T04:35:30.970 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:35:30.970 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:35:30.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.979+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.981+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.981+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.981+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.981+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.981+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.981+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.981+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.989+0000 7fe7ccf4d7c0 20 RGWRados::pool_iterate: got zone_info.e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 2024-01-16T04:35:30.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.989+0000 7fe7ccf4d7c0 20 RGWRados::pool_iterate: got zonegroup_info.78faec5d-171d-49c3-930a-c335c902808d 2024-01-16T04:35:30.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.989+0000 7fe7ccf4d7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:35:30.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.989+0000 7fe7ccf4d7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:35:30.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.989+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.989+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.989+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.989+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:35:30.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.989+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.990+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.990+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.990+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:35:30.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.990+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.990+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.990+0000 7fe7ccf4d7c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:35:30.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.990+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.990+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.990+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.991+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.991+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.991+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:35:30.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.991+0000 7fe7ccf4d7c0 20 zonegroup default 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.991+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.991+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.991+0000 7fe7ccf4d7c0 10 Cannot find current period zone using local zone 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.991+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.991+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.991+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.991+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.991+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.992+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.992+0000 7fe7ccf4d7c0 20 zone default found 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.992+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.992+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.992+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:30.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.992+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:35:30.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.992+0000 7fe7ccf4d7c0 20 started sync module instance, tier type = 2024-01-16T04:35:30.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:30.992+0000 7fe7ccf4d7c0 20 started zone id=e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 (name=default) with tier type = 2024-01-16T04:35:30.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.001+0000 7fe7ccf4d7c0 20 add_watcher() i=0 2024-01-16T04:35:30.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.001+0000 7fe7ccf4d7c0 20 add_watcher() i=1 2024-01-16T04:35:30.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.001+0000 7fe7ccf4d7c0 20 add_watcher() i=2 2024-01-16T04:35:30.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.001+0000 7fe7ccf4d7c0 20 add_watcher() i=3 2024-01-16T04:35:30.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.001+0000 7fe7ccf4d7c0 20 add_watcher() i=4 2024-01-16T04:35:30.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.001+0000 7fe7ccf4d7c0 20 add_watcher() i=5 2024-01-16T04:35:30.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.001+0000 7fe7ccf4d7c0 20 add_watcher() i=6 2024-01-16T04:35:30.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.002+0000 7fe7ccf4d7c0 20 add_watcher() i=7 2024-01-16T04:35:30.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.002+0000 7fe7ccf4d7c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.004+0000 7fe789fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.004+0000 7fe7ccf4d7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.004+0000 7fe7ccf4d7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:35:31.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.008+0000 7fe7ccf4d7c0 5 note: GC not initialized 2024-01-16T04:35:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe700ff9700 20 reqs_thread_entry: start 2024-01-16T04:35:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe6f67fc700 20 reqs_thread_entry: start 2024-01-16T04:35:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe7ccf4d7c0 20 init_complete bucket index max shards: 11 2024-01-16T04:35:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe7ccf4d7c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:35:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe6f4ff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:36:01 2024 2024-01-16T04:35:31.006 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:35:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe6dbfff700 20 reqs_thread_entry: start 2024-01-16T04:35:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe7ccf4d7c0 20 get_system_obj_state: rctx=0x7ffc49a0ca08 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x564c5d8f9d50 s->prefetch_data=0 2024-01-16T04:35:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe7ccf4d7c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-01-16T04:35:31.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe7ccf4d7c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2024-01-16T04:35:31.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe7ccf4d7c0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:35:31.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe7ccf4d7c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:35:31.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe7ccf4d7c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:35:31.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.009+0000 7fe7ccf4d7c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:35:31.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.010+0000 7fe7ccf4d7c0 20 rados->read ofs=0 len=0 2024-01-16T04:35:31.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.010+0000 7fe7ccf4d7c0 20 rados_obj.operate() r=0 bl.length=514 2024-01-16T04:35:31.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.010+0000 7fe7ccf4d7c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x11 2024-01-16T04:35:31.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.010+0000 7fe7ccf4d7c0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:35:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.012+0000 7fe7ccf4d7c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-01-16T04:35:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.012+0000 7fe7ccf4d7c0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:35:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.012+0000 7fe7ccf4d7c0 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-01-16T04:35:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.013+0000 7fe7b60a9700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084293 cookie 94885988526768 notifier 4229 bl.length()=728 2024-01-16T04:35:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.013+0000 7fe7b60a9700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-01-16T04:35:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.013+0000 7fe7b60a9700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:35:31.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.015+0000 7fe7ccf4d7c0 20 remove_watcher() i=0 2024-01-16T04:35:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.015+0000 7fe7ccf4d7c0 2 removed watcher, disabling cache 2024-01-16T04:35:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.016+0000 7fe7ccf4d7c0 20 remove_watcher() i=1 2024-01-16T04:35:31.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.017+0000 7fe7ccf4d7c0 20 remove_watcher() i=2 2024-01-16T04:35:31.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.018+0000 7fe7ccf4d7c0 20 remove_watcher() i=3 2024-01-16T04:35:31.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.019+0000 7fe7ccf4d7c0 20 remove_watcher() i=4 2024-01-16T04:35:31.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.020+0000 7fe7ccf4d7c0 20 remove_watcher() i=5 2024-01-16T04:35:31.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.021+0000 7fe7ccf4d7c0 20 remove_watcher() i=6 2024-01-16T04:35:31.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.022+0000 7fe7ccf4d7c0 20 remove_watcher() i=7 2024-01-16T04:35:31.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:35:31.022+0000 7fe789fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:35:31.023 INFO:tasks.s3tests:Configuring s3-tests... 2024-01-16T04:35:31.023 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest/s3-tests && ./bootstrap 2024-01-16T04:35:40.827 INFO:teuthology.orchestra.run.smithi038.stderr:./bootstrap: Missing required RPM packages: libevent-devel libffi-devel libxml2-devel libxslt-devel zlib-devel. 2024-01-16T04:35:41.702 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:04:09 ago on Tue 16 Jan 2024 04:31:32 AM UTC. 2024-01-16T04:35:43.541 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-01-16T04:35:43.543 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:35:43.543 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout: libevent-devel x86_64 2.1.8-5.el8 appstream 104 k 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout: libffi-devel x86_64 3.1-24.el8 baseos 29 k 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout: libxml2-devel x86_64 2.9.7-18.el8 appstream 1.1 M 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout: libxslt-devel x86_64 1.1.32-6.el8 appstream 323 k 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel x86_64 1.2.11-25.el8 baseos 59 k 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout: libxml2 x86_64 2.9.7-18.el8 baseos 697 k 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout: python3-libxml2 x86_64 2.9.7-18.el8 baseos 238 k 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout: zlib x86_64 1.2.11-25.el8 baseos 103 k 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-01-16T04:35:43.544 INFO:teuthology.orchestra.run.smithi038.stdout: cmake-filesystem x86_64 3.26.5-2.el8 appstream 52 k 2024-01-16T04:35:43.545 INFO:teuthology.orchestra.run.smithi038.stdout: libgcrypt-devel x86_64 1.8.5-7.el8 baseos 150 k 2024-01-16T04:35:43.545 INFO:teuthology.orchestra.run.smithi038.stdout: libgpg-error-devel x86_64 1.31-1.el8 baseos 64 k 2024-01-16T04:35:43.545 INFO:teuthology.orchestra.run.smithi038.stdout: xz-devel x86_64 5.2.4-4.el8 baseos 63 k 2024-01-16T04:35:43.545 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:43.545 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-01-16T04:35:43.545 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-01-16T04:35:43.545 INFO:teuthology.orchestra.run.smithi038.stdout:Install 9 Packages 2024-01-16T04:35:43.545 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 3 Packages 2024-01-16T04:35:43.545 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:43.546 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 2.9 M 2024-01-16T04:35:43.546 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-01-16T04:35:43.900 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): cmake-filesystem-3.26.5-2.el8.x86_64.rp 203 kB/s | 52 kB 00:00 2024-01-16T04:35:44.050 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): libevent-devel-2.1.8-5.el8.x86_64.rpm 256 kB/s | 104 kB 00:00 2024-01-16T04:35:44.175 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): libffi-devel-3.1-24.el8.x86_64.rpm 233 kB/s | 29 kB 00:00 2024-01-16T04:35:44.225 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): libxslt-devel-1.1.32-6.el8.x86_64.rpm 991 kB/s | 323 kB 00:00 2024-01-16T04:35:44.292 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): libgcrypt-devel-1.8.5-7.el8.x86_64.rpm 1.3 MB/s | 150 kB 00:00 2024-01-16T04:35:44.326 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): libgpg-error-devel-1.31-1.el8.x86_64.rp 637 kB/s | 64 kB 00:00 2024-01-16T04:35:44.351 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): xz-devel-5.2.4-4.el8.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-01-16T04:35:44.377 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): zlib-devel-1.2.11-25.el8.x86_64.rpm 1.1 MB/s | 59 kB 00:00 2024-01-16T04:35:44.460 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): python3-libxml2-2.9.7-18.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2024-01-16T04:35:44.527 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): libxml2-2.9.7-18.el8.x86_64.rpm 3.9 MB/s | 697 kB 00:00 2024-01-16T04:35:44.552 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): zlib-1.2.11-25.el8.x86_64.rpm 1.1 MB/s | 103 kB 00:00 2024-01-16T04:35:45.078 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): libxml2-devel-2.9.7-18.el8.x86_64.rpm 792 kB/s | 1.1 MB 00:01 2024-01-16T04:35:45.078 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:35:45.078 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.9 MB/s | 2.9 MB 00:01 2024-01-16T04:35:45.202 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-01-16T04:35:45.350 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-01-16T04:35:45.350 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-01-16T04:35:45.504 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-01-16T04:35:45.507 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-01-16T04:35:46.179 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-01-16T04:35:46.490 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : zlib-1.2.11-25.el8.x86_64 1/15 2024-01-16T04:35:46.750 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : libxml2-2.9.7-18.el8.x86_64 2/15 2024-01-16T04:35:46.869 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 3/15 2024-01-16T04:35:46.895 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 3/15 2024-01-16T04:35:47.099 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgcrypt-devel-1.8.5-7.el8.x86_64 4/15 2024-01-16T04:35:47.112 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libgcrypt-devel-1.8.5-7.el8.x86_64 4/15 2024-01-16T04:35:47.353 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : zlib-devel-1.2.11-25.el8.x86_64 5/15 2024-01-16T04:35:47.521 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xz-devel-5.2.4-4.el8.x86_64 6/15 2024-01-16T04:35:48.003 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cmake-filesystem-3.26.5-2.el8.x86_64 7/15 2024-01-16T04:35:48.350 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libxml2-devel-2.9.7-18.el8.x86_64 8/15 2024-01-16T04:35:48.617 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 9/15 2024-01-16T04:35:48.893 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : python3-libxml2-2.9.7-18.el8.x86_64 10/15 2024-01-16T04:35:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libffi-devel-3.1-24.el8.x86_64 11/15 2024-01-16T04:35:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libffi-devel-3.1-24.el8.x86_64 11/15 2024-01-16T04:35:49.246 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 12/15 2024-01-16T04:35:49.439 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : python3-libxml2-2.9.7-16.el8.x86_64 13/15 2024-01-16T04:35:49.595 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : libxml2-2.9.7-16.el8.x86_64 14/15 2024-01-16T04:35:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : zlib-1.2.11-21.el8.x86_64 15/15 2024-01-16T04:35:50.331 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: zlib-1.2.11-21.el8.x86_64 15/15 2024-01-16T04:35:50.332 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cmake-filesystem-3.26.5-2.el8.x86_64 1/15 2024-01-16T04:35:50.332 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 2/15 2024-01-16T04:35:50.332 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libxml2-devel-2.9.7-18.el8.x86_64 3/15 2024-01-16T04:35:50.332 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 4/15 2024-01-16T04:35:50.332 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libffi-devel-3.1-24.el8.x86_64 5/15 2024-01-16T04:35:50.332 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgcrypt-devel-1.8.5-7.el8.x86_64 6/15 2024-01-16T04:35:50.332 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 7/15 2024-01-16T04:35:50.332 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xz-devel-5.2.4-4.el8.x86_64 8/15 2024-01-16T04:35:50.332 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : zlib-devel-1.2.11-25.el8.x86_64 9/15 2024-01-16T04:35:50.332 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libxml2-2.9.7-18.el8.x86_64 10/15 2024-01-16T04:35:50.332 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libxml2-2.9.7-16.el8.x86_64 11/15 2024-01-16T04:35:50.332 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-libxml2-2.9.7-18.el8.x86_64 12/15 2024-01-16T04:35:50.333 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-libxml2-2.9.7-16.el8.x86_64 13/15 2024-01-16T04:35:50.333 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : zlib-1.2.11-25.el8.x86_64 14/15 2024-01-16T04:35:51.418 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : zlib-1.2.11-21.el8.x86_64 15/15 2024-01-16T04:35:51.418 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:51.419 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-01-16T04:35:51.419 INFO:teuthology.orchestra.run.smithi038.stdout: libxml2-2.9.7-18.el8.x86_64 python3-libxml2-2.9.7-18.el8.x86_64 2024-01-16T04:35:51.419 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-1.2.11-25.el8.x86_64 2024-01-16T04:35:51.419 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-01-16T04:35:51.419 INFO:teuthology.orchestra.run.smithi038.stdout: cmake-filesystem-3.26.5-2.el8.x86_64 libevent-devel-2.1.8-5.el8.x86_64 2024-01-16T04:35:51.419 INFO:teuthology.orchestra.run.smithi038.stdout: libffi-devel-3.1-24.el8.x86_64 libgcrypt-devel-1.8.5-7.el8.x86_64 2024-01-16T04:35:51.419 INFO:teuthology.orchestra.run.smithi038.stdout: libgpg-error-devel-1.31-1.el8.x86_64 libxml2-devel-2.9.7-18.el8.x86_64 2024-01-16T04:35:51.419 INFO:teuthology.orchestra.run.smithi038.stdout: libxslt-devel-1.1.32-6.el8.x86_64 xz-devel-5.2.4-4.el8.x86_64 2024-01-16T04:35:51.419 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel-1.2.11-25.el8.x86_64 2024-01-16T04:35:51.419 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:35:51.419 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-01-16T04:35:52.110 INFO:teuthology.orchestra.run.smithi038.stdout:Already using interpreter /usr/bin/python3.6 2024-01-16T04:35:52.110 INFO:teuthology.orchestra.run.smithi038.stdout:Using base prefix '/usr' 2024-01-16T04:35:52.110 INFO:teuthology.orchestra.run.smithi038.stdout:New python executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python3.6 2024-01-16T04:35:52.110 INFO:teuthology.orchestra.run.smithi038.stdout:Also creating executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python 2024-01-16T04:35:55.894 INFO:teuthology.orchestra.run.smithi038.stdout:Installing setuptools, pip, wheel...done. 2024-01-16T04:35:56.255 INFO:teuthology.orchestra.run.smithi038.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.6/site-packages (21.3.1) 2024-01-16T04:35:57.615 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting setuptools==44.1.0 2024-01-16T04:35:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2024-01-16T04:35:57.819 INFO:teuthology.orchestra.run.smithi038.stdout:Installing collected packages: setuptools 2024-01-16T04:35:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: Attempting uninstall: setuptools 2024-01-16T04:35:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: Found existing installation: setuptools 59.6.0 2024-01-16T04:35:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: Uninstalling setuptools-59.6.0: 2024-01-16T04:35:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: Successfully uninstalled setuptools-59.6.0 2024-01-16T04:35:58.111 INFO:teuthology.orchestra.run.smithi038.stdout:Successfully installed setuptools-44.1.0 2024-01-16T04:35:58.731 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting PyYAML 2024-01-16T04:35:58.790 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB) 2024-01-16T04:35:58.967 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting nose>=1.0.0 2024-01-16T04:35:58.985 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2024-01-16T04:35:59.098 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting boto>=2.6.0 2024-01-16T04:35:59.120 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2024-01-16T04:36:00.231 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting boto3>=1.0.0 2024-01-16T04:36:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2024-01-16T04:36:01.533 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting botocore<1.28.0 2024-01-16T04:36:01.581 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2024-01-16T04:36:01.905 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting munch>=2.0.0 2024-01-16T04:36:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading munch-4.0.0-py2.py3-none-any.whl (9.9 kB) 2024-01-16T04:36:02.459 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting gevent>=1.0 2024-01-16T04:36:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading gevent-22.10.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB) 2024-01-16T04:36:02.705 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting isodate>=0.4.4 2024-01-16T04:36:02.722 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2024-01-16T04:36:02.856 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting requests>=2.23.0 2024-01-16T04:36:02.875 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-01-16T04:36:03.080 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting pytz>=2011k 2024-01-16T04:36:03.098 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) 2024-01-16T04:36:03.205 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting httplib2 2024-01-16T04:36:03.222 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading httplib2-0.22.0-py3-none-any.whl (96 kB) 2024-01-16T04:36:03.857 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting lxml 2024-01-16T04:36:03.878 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading lxml-5.1.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.1 MB) 2024-01-16T04:36:04.348 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting jmespath<2.0.0,>=0.7.1 2024-01-16T04:36:04.365 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2024-01-16T04:36:04.503 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2024-01-16T04:36:04.529 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2024-01-16T04:36:04.633 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting python-dateutil<3.0.0,>=2.1 2024-01-16T04:36:04.657 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2024-01-16T04:36:04.830 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting urllib3<1.27,>=1.25.4 2024-01-16T04:36:04.872 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2024-01-16T04:36:05.097 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting importlib-metadata>=1.7.0 2024-01-16T04:36:05.114 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB) 2024-01-16T04:36:05.184 INFO:teuthology.orchestra.run.smithi038.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (from gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2024-01-16T04:36:05.240 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting zope.event 2024-01-16T04:36:05.258 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB) 2024-01-16T04:36:05.676 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting zope.interface 2024-01-16T04:36:05.706 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253 kB) 2024-01-16T04:36:06.288 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting greenlet>=2.0.0 2024-01-16T04:36:06.308 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading greenlet-2.0.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (564 kB) 2024-01-16T04:36:06.416 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting six 2024-01-16T04:36:06.434 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2024-01-16T04:36:06.580 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting idna<4,>=2.5 2024-01-16T04:36:06.600 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading idna-3.6-py3-none-any.whl (61 kB) 2024-01-16T04:36:06.941 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting charset-normalizer~=2.0.0 2024-01-16T04:36:06.961 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-01-16T04:36:07.050 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting certifi>=2017.4.17 2024-01-16T04:36:07.073 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading certifi-2023.11.17-py3-none-any.whl (162 kB) 2024-01-16T04:36:07.249 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2024-01-16T04:36:07.267 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB) 2024-01-16T04:36:07.456 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting zipp>=0.5 2024-01-16T04:36:07.478 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB) 2024-01-16T04:36:07.579 INFO:teuthology.orchestra.run.smithi038.stdout:Collecting typing-extensions>=3.6.4 2024-01-16T04:36:07.602 INFO:teuthology.orchestra.run.smithi038.stdout: Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB) 2024-01-16T04:36:08.168 INFO:teuthology.orchestra.run.smithi038.stdout:Installing collected packages: six, urllib3, python-dateutil, jmespath, zipp, typing-extensions, botocore, zope.interface, zope.event, s3transfer, pyparsing, importlib-metadata, idna, greenlet, charset-normalizer, certifi, requests, PyYAML, pytz, nose, munch, lxml, isodate, httplib2, gevent, boto3, boto 2024-01-16T04:36:12.126 INFO:teuthology.orchestra.run.smithi038.stdout:Successfully installed PyYAML-6.0.1 boto-2.49.0 boto3-1.23.10 botocore-1.26.10 certifi-2023.11.17 charset-normalizer-2.0.12 gevent-22.10.2 greenlet-2.0.2 httplib2-0.22.0 idna-3.6 importlib-metadata-4.8.3 isodate-0.6.1 jmespath-0.10.0 lxml-5.1.0 munch-4.0.0 nose-1.3.7 pyparsing-3.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 requests-2.27.1 s3transfer-0.5.2 six-1.16.0 typing-extensions-4.1.1 urllib3-1.26.18 zipp-3.6.0 zope.event-4.6 zope.interface-5.5.2 2024-01-16T04:36:12.453 INFO:teuthology.orchestra.run.smithi038.stdout:running develop 2024-01-16T04:36:12.476 INFO:teuthology.orchestra.run.smithi038.stdout:running egg_info 2024-01-16T04:36:12.476 INFO:teuthology.orchestra.run.smithi038.stdout:creating s3tests.egg-info 2024-01-16T04:36:12.476 INFO:teuthology.orchestra.run.smithi038.stdout:writing s3tests.egg-info/PKG-INFO 2024-01-16T04:36:12.476 INFO:teuthology.orchestra.run.smithi038.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2024-01-16T04:36:12.476 INFO:teuthology.orchestra.run.smithi038.stdout:writing requirements to s3tests.egg-info/requires.txt 2024-01-16T04:36:12.476 INFO:teuthology.orchestra.run.smithi038.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2024-01-16T04:36:12.476 INFO:teuthology.orchestra.run.smithi038.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2024-01-16T04:36:12.487 INFO:teuthology.orchestra.run.smithi038.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2024-01-16T04:36:12.488 INFO:teuthology.orchestra.run.smithi038.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2024-01-16T04:36:12.509 INFO:teuthology.orchestra.run.smithi038.stdout:running build_ext 2024-01-16T04:36:12.509 INFO:teuthology.orchestra.run.smithi038.stdout:Creating /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/s3tests.egg-link (link to .) 2024-01-16T04:36:12.509 INFO:teuthology.orchestra.run.smithi038.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2024-01-16T04:36:12.509 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.510 INFO:teuthology.orchestra.run.smithi038.stdout:Installed /home/ubuntu/cephtest/s3-tests 2024-01-16T04:36:12.510 INFO:teuthology.orchestra.run.smithi038.stdout:Processing dependencies for s3tests==0.0.1 2024-01-16T04:36:12.628 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for isodate==0.6.1 2024-01-16T04:36:12.628 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: isodate 0.6.1 2024-01-16T04:36:12.628 INFO:teuthology.orchestra.run.smithi038.stdout:Adding isodate 0.6.1 to easy-install.pth file 2024-01-16T04:36:12.628 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.628 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.629 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for gevent==22.10.2 2024-01-16T04:36:12.629 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: gevent 22.10.2 2024-01-16T04:36:12.629 INFO:teuthology.orchestra.run.smithi038.stdout:Adding gevent 22.10.2 to easy-install.pth file 2024-01-16T04:36:12.629 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.629 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.630 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for munch==4.0.0 2024-01-16T04:36:12.630 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: munch 4.0.0 2024-01-16T04:36:12.630 INFO:teuthology.orchestra.run.smithi038.stdout:Adding munch 4.0.0 to easy-install.pth file 2024-01-16T04:36:12.630 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.630 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.631 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for PyYAML==6.0.1 2024-01-16T04:36:12.631 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: PyYAML 6.0.1 2024-01-16T04:36:12.631 INFO:teuthology.orchestra.run.smithi038.stdout:Adding PyYAML 6.0.1 to easy-install.pth file 2024-01-16T04:36:12.631 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.631 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.633 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for boto3==1.23.10 2024-01-16T04:36:12.633 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: boto3 1.23.10 2024-01-16T04:36:12.633 INFO:teuthology.orchestra.run.smithi038.stdout:Adding boto3 1.23.10 to easy-install.pth file 2024-01-16T04:36:12.633 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.633 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.633 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for boto==2.49.0 2024-01-16T04:36:12.633 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: boto 2.49.0 2024-01-16T04:36:12.634 INFO:teuthology.orchestra.run.smithi038.stdout:Adding boto 2.49.0 to easy-install.pth file 2024-01-16T04:36:12.634 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.634 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.639 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for six==1.16.0 2024-01-16T04:36:12.639 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: six 1.16.0 2024-01-16T04:36:12.639 INFO:teuthology.orchestra.run.smithi038.stdout:Adding six 1.16.0 to easy-install.pth file 2024-01-16T04:36:12.639 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.639 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.639 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for setuptools==44.1.0 2024-01-16T04:36:12.640 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: setuptools 44.1.0 2024-01-16T04:36:12.640 INFO:teuthology.orchestra.run.smithi038.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2024-01-16T04:36:12.640 INFO:teuthology.orchestra.run.smithi038.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2024-01-16T04:36:12.640 INFO:teuthology.orchestra.run.smithi038.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2024-01-16T04:36:12.640 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.640 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.640 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for greenlet==2.0.2 2024-01-16T04:36:12.641 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: greenlet 2.0.2 2024-01-16T04:36:12.641 INFO:teuthology.orchestra.run.smithi038.stdout:Adding greenlet 2.0.2 to easy-install.pth file 2024-01-16T04:36:12.641 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.641 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.641 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for zope.interface==5.5.2 2024-01-16T04:36:12.641 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: zope.interface 5.5.2 2024-01-16T04:36:12.642 INFO:teuthology.orchestra.run.smithi038.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2024-01-16T04:36:12.642 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.642 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.642 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for zope.event==4.6 2024-01-16T04:36:12.642 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: zope.event 4.6 2024-01-16T04:36:12.642 INFO:teuthology.orchestra.run.smithi038.stdout:Adding zope.event 4.6 to easy-install.pth file 2024-01-16T04:36:12.643 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.643 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.649 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for importlib-metadata==4.8.3 2024-01-16T04:36:12.649 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: importlib-metadata 4.8.3 2024-01-16T04:36:12.650 INFO:teuthology.orchestra.run.smithi038.stdout:Adding importlib-metadata 4.8.3 to easy-install.pth file 2024-01-16T04:36:12.650 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.650 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.650 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for botocore==1.26.10 2024-01-16T04:36:12.650 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: botocore 1.26.10 2024-01-16T04:36:12.650 INFO:teuthology.orchestra.run.smithi038.stdout:Adding botocore 1.26.10 to easy-install.pth file 2024-01-16T04:36:12.651 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.651 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.651 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for s3transfer==0.5.2 2024-01-16T04:36:12.651 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: s3transfer 0.5.2 2024-01-16T04:36:12.651 INFO:teuthology.orchestra.run.smithi038.stdout:Adding s3transfer 0.5.2 to easy-install.pth file 2024-01-16T04:36:12.651 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.652 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.652 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for jmespath==0.10.0 2024-01-16T04:36:12.652 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: jmespath 0.10.0 2024-01-16T04:36:12.652 INFO:teuthology.orchestra.run.smithi038.stdout:Adding jmespath 0.10.0 to easy-install.pth file 2024-01-16T04:36:12.652 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.652 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.653 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for typing-extensions==4.1.1 2024-01-16T04:36:12.653 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: typing-extensions 4.1.1 2024-01-16T04:36:12.653 INFO:teuthology.orchestra.run.smithi038.stdout:Adding typing-extensions 4.1.1 to easy-install.pth file 2024-01-16T04:36:12.653 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.653 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.653 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for zipp==3.6.0 2024-01-16T04:36:12.654 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: zipp 3.6.0 2024-01-16T04:36:12.654 INFO:teuthology.orchestra.run.smithi038.stdout:Adding zipp 3.6.0 to easy-install.pth file 2024-01-16T04:36:12.654 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.654 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.654 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for python-dateutil==2.8.2 2024-01-16T04:36:12.654 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: python-dateutil 2.8.2 2024-01-16T04:36:12.655 INFO:teuthology.orchestra.run.smithi038.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2024-01-16T04:36:12.655 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.655 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.655 INFO:teuthology.orchestra.run.smithi038.stdout:Searching for urllib3==1.26.18 2024-01-16T04:36:12.655 INFO:teuthology.orchestra.run.smithi038.stdout:Best match: urllib3 1.26.18 2024-01-16T04:36:12.655 INFO:teuthology.orchestra.run.smithi038.stdout:Adding urllib3 1.26.18 to easy-install.pth file 2024-01-16T04:36:12.655 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:36:12.656 INFO:teuthology.orchestra.run.smithi038.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:36:12.656 INFO:teuthology.orchestra.run.smithi038.stdout:Finished processing dependencies for s3tests==0.0.1 2024-01-16T04:36:12.666 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:36:12.666 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2024-01-16T04:36:12.684 INFO:tasks.s3tests:Configuring boto... 2024-01-16T04:36:12.684 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-01-16T04:36:12.685 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/boto.cfg 2024-01-16T04:36:12.743 DEBUG:teuthology.orchestra.run.smithi038:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain' 2024-01-16T04:36:13.210 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2024-01-16T04:36:13.262 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2024-01-16T04:36:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2024-01-16T04:36:13.365 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2024-01-16T04:36:13.416 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2024-01-16T04:36:13.425 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2024-01-16T04:36:13.480 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2024-01-16T04:36:13.489 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2024-01-16T04:36:13.531 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2024-01-16T04:36:13.573 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2024-01-16T04:36:13.627 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2024-01-16T04:36:13.678 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2024-01-16T04:36:13.729 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2024-01-16T04:36:13.780 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2024-01-16T04:36:13.781 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2024-01-16T04:36:13.826 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2024-01-16T04:36:13.826 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2024-01-16T04:36:13.827 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2024-01-16T04:36:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2024-01-16T04:36:13.829 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2024-01-16T04:36:13.829 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2024-01-16T04:36:13.829 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2024-01-16T04:36:13.829 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2024-01-16T04:36:13.829 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2024-01-16T04:36:13.829 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2024-01-16T04:36:13.829 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2024-01-16T04:36:13.829 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2024-01-16T04:36:13.971 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2024-01-16T04:36:13.971 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_lifecycle_transition ... SKIP 2024-01-16T04:36:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2024-01-16T04:36:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2024-01-16T04:36:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2024-01-16T04:36:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2024-01-16T04:36:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2024-01-16T04:36:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2024-01-16T04:36:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2024-01-16T04:36:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2024-01-16T04:36:14.066 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2024-01-16T04:36:14.108 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2024-01-16T04:36:15.746 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2024-01-16T04:36:15.839 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2024-01-16T04:36:15.906 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2024-01-16T04:36:15.967 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_enc ... ok 2024-01-16T04:36:15.988 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2024-01-16T04:36:16.068 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2024-01-16T04:36:16.069 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_pubilc_errordoc ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2024-01-16T04:36:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2024-01-16T04:36:16.071 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2024-01-16T04:36:16.071 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2024-01-16T04:36:16.071 INFO:teuthology.orchestra.run.smithi038.stderr:SKIP 2024-01-16T04:36:19.112 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests.functional.test_utils.test_generate ... ok 2024-01-16T04:36:19.112 INFO:teuthology.orchestra.run.smithi038.stderr:/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2024-01-16T04:36:19.112 INFO:teuthology.orchestra.run.smithi038.stderr: warnings.warn(warning, PythonDeprecationWarning) 2024-01-16T04:36:19.136 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2024-01-16T04:36:26.426 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2024-01-16T04:36:26.456 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2024-01-16T04:36:26.500 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2024-01-16T04:36:26.542 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2024-01-16T04:36:26.584 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2024-01-16T04:36:26.624 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2024-01-16T04:36:26.672 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2024-01-16T04:36:26.712 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2024-01-16T04:36:26.739 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2024-01-16T04:36:26.766 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2024-01-16T04:36:26.791 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2024-01-16T04:36:26.806 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2024-01-16T04:36:26.822 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2024-01-16T04:36:26.831 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2024-01-16T04:36:26.855 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2024-01-16T04:36:26.887 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2024-01-16T04:36:26.919 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2024-01-16T04:36:26.944 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2024-01-16T04:36:26.968 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2024-01-16T04:36:26.993 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2024-01-16T04:36:27.016 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2024-01-16T04:36:27.040 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2024-01-16T04:36:27.057 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2024-01-16T04:36:27.072 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2024-01-16T04:36:27.080 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2024-01-16T04:36:27.088 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2024-01-16T04:36:27.097 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2024-01-16T04:36:27.104 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2024-01-16T04:36:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2024-01-16T04:36:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2024-01-16T04:36:27.283 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2024-01-16T04:36:27.376 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2024-01-16T04:36:27.468 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2024-01-16T04:36:27.557 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2024-01-16T04:36:27.612 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2024-01-16T04:36:27.693 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2024-01-16T04:36:27.747 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2024-01-16T04:36:27.802 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2024-01-16T04:36:27.924 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2024-01-16T04:36:28.047 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2024-01-16T04:36:28.082 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2024-01-16T04:36:28.117 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2024-01-16T04:36:28.172 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2024-01-16T04:36:28.226 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2024-01-16T04:36:28.350 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2024-01-16T04:36:28.496 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2024-01-16T04:36:28.552 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2024-01-16T04:36:28.609 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2024-01-16T04:36:28.664 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2024-01-16T04:36:28.721 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2024-01-16T04:36:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2024-01-16T04:36:28.832 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2024-01-16T04:36:28.885 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2024-01-16T04:36:28.940 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2024-01-16T04:36:28.996 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2024-01-16T04:36:29.052 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2024-01-16T04:36:29.106 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2024-01-16T04:36:29.162 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2024-01-16T04:36:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2024-01-16T04:36:29.287 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2024-01-16T04:36:29.334 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2024-01-16T04:36:29.389 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2024-01-16T04:36:29.446 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2024-01-16T04:36:36.402 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2024-01-16T04:36:36.475 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2024-01-16T04:36:36.523 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2024-01-16T04:36:36.573 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2024-01-16T04:36:36.624 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2024-01-16T04:36:36.673 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2024-01-16T04:36:36.721 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2024-01-16T04:36:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2024-01-16T04:36:36.818 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2024-01-16T04:36:36.867 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2024-01-16T04:36:36.915 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2024-01-16T04:36:36.963 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2024-01-16T04:36:37.013 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2024-01-16T04:36:37.068 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2024-01-16T04:36:37.123 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2024-01-16T04:36:37.178 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2024-01-16T04:36:37.233 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2024-01-16T04:36:37.314 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2024-01-16T04:36:37.368 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2024-01-16T04:36:37.424 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2024-01-16T04:36:37.480 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2024-01-16T04:36:37.534 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2024-01-16T04:36:37.589 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2024-01-16T04:36:37.690 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2024-01-16T04:36:37.791 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2024-01-16T04:36:37.846 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2024-01-16T04:36:37.900 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2024-01-16T04:36:37.955 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2024-01-16T04:36:38.011 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2024-01-16T04:36:38.255 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_account_usage ... ok 2024-01-16T04:36:38.289 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_head_bucket_usage ... ok 2024-01-16T04:36:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2024-01-16T04:36:39.097 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2024-01-16T04:36:39.155 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2024-01-16T04:36:39.211 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2024-01-16T04:36:39.267 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2024-01-16T04:36:39.321 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2024-01-16T04:36:39.422 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken ... ok 2024-01-16T04:36:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_both_continuationtoken_startafter ... ok 2024-01-16T04:36:39.585 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2024-01-16T04:36:39.641 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2024-01-16T04:36:39.699 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2024-01-16T04:36:39.756 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2024-01-16T04:36:39.838 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2024-01-16T04:36:39.892 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2024-01-16T04:36:40.124 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2024-01-16T04:36:40.439 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2024-01-16T04:36:40.473 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2024-01-16T04:36:40.506 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2024-01-16T04:36:40.531 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2024-01-16T04:36:40.556 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2024-01-16T04:36:40.565 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2024-01-16T04:36:40.576 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2024-01-16T04:36:40.584 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2024-01-16T04:36:40.626 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2024-01-16T04:36:40.785 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2024-01-16T04:36:40.795 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2024-01-16T04:36:40.890 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2024-01-16T04:36:40.917 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_read_not_exist ... ok 2024-01-16T04:36:40.945 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2024-01-16T04:36:41.171 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2024-01-16T04:36:41.397 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2024-01-16T04:36:41.430 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2024-01-16T04:36:41.460 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2024-01-16T04:36:41.494 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2024-01-16T04:36:41.526 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2024-01-16T04:36:41.683 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2024-01-16T04:36:41.716 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2024-01-16T04:36:41.749 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2024-01-16T04:36:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2024-01-16T04:36:41.841 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2024-01-16T04:36:41.913 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2024-01-16T04:36:41.982 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2024-01-16T04:36:42.016 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2024-01-16T04:36:42.083 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2024-01-16T04:36:42.106 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2024-01-16T04:36:42.128 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2024-01-16T04:36:42.153 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2024-01-16T04:36:42.225 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2024-01-16T04:36:42.257 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2024-01-16T04:36:42.283 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2024-01-16T04:36:42.311 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2024-01-16T04:36:42.343 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2024-01-16T04:36:42.414 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2024-01-16T04:36:42.439 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2024-01-16T04:36:42.461 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2024-01-16T04:36:42.484 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2024-01-16T04:36:42.507 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2024-01-16T04:36:42.530 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2024-01-16T04:36:42.552 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2024-01-16T04:36:42.587 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2024-01-16T04:36:42.611 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2024-01-16T04:36:42.634 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2024-01-16T04:36:42.658 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2024-01-16T04:36:42.681 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2024-01-16T04:36:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2024-01-16T04:36:42.728 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2024-01-16T04:36:42.751 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2024-01-16T04:36:42.775 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2024-01-16T04:36:42.799 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2024-01-16T04:36:42.823 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2024-01-16T04:36:42.846 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2024-01-16T04:36:42.870 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2024-01-16T04:36:42.943 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2024-01-16T04:36:43.021 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2024-01-16T04:36:43.057 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2024-01-16T04:36:43.131 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2024-01-16T04:36:43.230 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2024-01-16T04:36:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2024-01-16T04:36:44.357 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2024-01-16T04:36:44.435 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2024-01-16T04:36:44.560 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2024-01-16T04:36:44.727 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2024-01-16T04:36:44.850 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2024-01-16T04:36:44.962 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2024-01-16T04:36:45.086 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2024-01-16T04:36:45.248 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2024-01-16T04:36:45.322 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2024-01-16T04:36:45.483 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2024-01-16T04:36:45.515 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2024-01-16T04:36:45.588 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2024-01-16T04:36:45.661 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2024-01-16T04:36:45.705 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2024-01-16T04:36:45.729 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2024-01-16T04:36:45.739 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_notexist ... ok 2024-01-16T04:36:45.801 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2024-01-16T04:36:45.833 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2024-01-16T04:36:45.867 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2024-01-16T04:36:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2024-01-16T04:36:45.933 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2024-01-16T04:36:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2024-01-16T04:36:46.026 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2024-01-16T04:36:46.137 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2024-01-16T04:36:46.217 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2024-01-16T04:36:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2024-01-16T04:36:46.277 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired_tenant ... ok 2024-01-16T04:36:46.308 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2024-01-16T04:36:46.338 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2024-01-16T04:36:46.370 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2024-01-16T04:36:46.407 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2024-01-16T04:36:46.448 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2024-01-16T04:36:46.476 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2024-01-16T04:36:46.508 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2024-01-16T04:36:46.518 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2024-01-16T04:36:46.528 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2024-01-16T04:36:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2024-01-16T04:36:46.549 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2024-01-16T04:36:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2024-01-16T04:36:46.637 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2024-01-16T04:36:46.663 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2024-01-16T04:36:46.689 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl ... FAIL 2024-01-16T04:36:46.712 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl ... FAIL 2024-01-16T04:36:46.736 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2024-01-16T04:36:46.797 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2024-01-16T04:36:46.907 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2024-01-16T04:36:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2024-01-16T04:36:47.034 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2024-01-16T04:36:47.107 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2024-01-16T04:36:47.180 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2024-01-16T04:36:47.303 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2024-01-16T04:36:47.377 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2024-01-16T04:36:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2024-01-16T04:36:47.577 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2024-01-16T04:36:47.702 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2024-01-16T04:36:47.796 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2024-01-16T04:36:47.872 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2024-01-16T04:36:47.898 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2024-01-16T04:36:48.021 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2024-01-16T04:36:48.151 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2024-01-16T04:36:48.273 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2024-01-16T04:36:48.399 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2024-01-16T04:36:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2024-01-16T04:36:48.646 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2024-01-16T04:36:48.763 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2024-01-16T04:36:48.873 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2024-01-16T04:36:48.989 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2024-01-16T04:36:49.105 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2024-01-16T04:36:49.138 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2024-01-16T04:36:49.245 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2024-01-16T04:36:49.321 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2024-01-16T04:36:49.401 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2024-01-16T04:36:49.478 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2024-01-16T04:36:49.513 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_not_exist ... ok 2024-01-16T04:36:49.642 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2024-01-16T04:36:49.875 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2024-01-16T04:36:50.083 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2024-01-16T04:36:50.290 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2024-01-16T04:36:50.498 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2024-01-16T04:36:50.709 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2024-01-16T04:36:50.918 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2024-01-16T04:36:51.128 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2024-01-16T04:36:51.339 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2024-01-16T04:36:51.582 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2024-01-16T04:36:51.750 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2024-01-16T04:36:51.916 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2024-01-16T04:36:52.083 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2024-01-16T04:36:52.260 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2024-01-16T04:36:52.266 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2024-01-16T04:36:52.274 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2024-01-16T04:36:52.282 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2024-01-16T04:36:52.532 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2024-01-16T04:36:52.787 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2024-01-16T04:36:52.805 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2024-01-16T04:36:52.823 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2024-01-16T04:36:52.887 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2024-01-16T04:36:53.555 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2024-01-16T04:36:53.632 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2024-01-16T04:36:53.730 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2024-01-16T04:36:53.858 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2024-01-16T04:36:54.019 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2024-01-16T04:36:54.100 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2024-01-16T04:36:54.184 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2024-01-16T04:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2024-01-16T04:36:54.382 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2024-01-16T04:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2024-01-16T04:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2024-01-16T04:36:54.864 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2024-01-16T04:36:55.047 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2024-01-16T04:36:55.076 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2024-01-16T04:36:55.105 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2024-01-16T04:36:55.678 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2024-01-16T04:36:55.829 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2024-01-16T04:36:56.866 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2024-01-16T04:36:56.904 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2024-01-16T04:36:56.993 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2024-01-16T04:36:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2024-01-16T04:36:57.354 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2024-01-16T04:36:57.576 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2024-01-16T04:36:58.539 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2024-01-16T04:36:59.220 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2024-01-16T04:37:00.727 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2024-01-16T04:37:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2024-01-16T04:37:05.715 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2024-01-16T04:37:07.095 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2024-01-16T04:37:07.232 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2024-01-16T04:37:07.566 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2024-01-16T04:37:07.894 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2024-01-16T04:37:08.092 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2024-01-16T04:37:08.167 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2024-01-16T04:37:08.460 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2024-01-16T04:37:08.546 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2024-01-16T04:37:08.633 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2024-01-16T04:37:08.660 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2024-01-16T04:37:08.783 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2024-01-16T04:37:11.903 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2024-01-16T04:37:14.950 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2024-01-16T04:37:17.993 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2024-01-16T04:37:18.045 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_cors_presigned_get_object ... ok 2024-01-16T04:37:18.086 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_cors_presigned_get_object_tenant ... ok 2024-01-16T04:37:18.211 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_set_bucket_tagging ... ok 2024-01-16T04:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2024-01-16T04:37:18.479 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2024-01-16T04:37:18.764 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2024-01-16T04:37:18.907 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2024-01-16T04:37:19.160 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2024-01-16T04:37:19.539 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2024-01-16T04:37:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2024-01-16T04:37:19.918 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2024-01-16T04:37:20.361 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2024-01-16T04:37:20.485 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2024-01-16T04:37:20.655 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2024-01-16T04:37:20.928 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2024-01-16T04:37:21.006 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2024-01-16T04:37:27.090 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2024-01-16T04:37:27.165 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2024-01-16T04:37:27.238 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2024-01-16T04:37:27.314 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2024-01-16T04:37:27.394 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2024-01-16T04:37:27.655 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2024-01-16T04:37:35.401 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2024-01-16T04:37:35.969 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2024-01-16T04:37:36.163 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2024-01-16T04:37:36.468 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2024-01-16T04:37:36.810 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2024-01-16T04:37:39.807 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2024-01-16T04:37:43.460 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2024-01-16T04:37:57.820 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2024-01-16T04:37:59.224 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2024-01-16T04:37:59.899 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2024-01-16T04:38:00.871 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2024-01-16T04:38:01.272 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2024-01-16T04:38:01.729 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2024-01-16T04:38:01.859 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2024-01-16T04:38:02.294 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2024-01-16T04:38:02.643 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2024-01-16T04:38:02.926 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... ok 2024-01-16T04:38:02.959 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2024-01-16T04:38:03.034 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2024-01-16T04:38:03.108 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2024-01-16T04:38:03.137 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2024-01-16T04:38:03.164 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2024-01-16T04:38:03.280 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2024-01-16T04:38:03.312 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2024-01-16T04:38:03.339 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2024-01-16T04:38:03.390 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2024-01-16T04:38:03.420 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2024-01-16T04:38:03.453 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2024-01-16T04:38:03.484 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2024-01-16T04:38:03.514 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2024-01-16T04:38:03.590 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2024-01-16T04:38:03.667 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2024-01-16T04:38:03.722 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2024-01-16T04:38:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2024-01-16T04:38:03.842 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2024-01-16T04:38:03.920 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2024-01-16T04:38:03.998 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2024-01-16T04:38:04.092 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2024-01-16T04:38:04.160 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2024-01-16T04:38:04.229 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2024-01-16T04:38:04.297 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2024-01-16T04:38:05.155 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2024-01-16T04:38:05.711 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2024-01-16T04:38:05.747 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2024-01-16T04:38:05.792 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2024-01-16T04:38:05.867 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2024-01-16T04:38:05.937 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2024-01-16T04:38:06.005 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2024-01-16T04:38:06.840 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2024-01-16T04:38:07.328 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2024-01-16T04:38:07.814 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2024-01-16T04:38:07.848 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2024-01-16T04:38:07.924 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2024-01-16T04:38:07.998 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2024-01-16T04:38:08.049 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2024-01-16T04:38:08.124 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2024-01-16T04:38:08.201 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2024-01-16T04:38:08.246 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2024-01-16T04:38:08.292 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2024-01-16T04:38:08.353 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2024-01-16T04:38:08.440 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2024-01-16T04:38:08.567 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2024-01-16T04:38:08.698 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2024-01-16T04:38:08.864 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2024-01-16T04:38:08.983 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2024-01-16T04:38:09.149 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2024-01-16T04:38:09.308 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2024-01-16T04:38:09.476 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2024-01-16T04:38:09.637 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2024-01-16T04:38:09.805 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2024-01-16T04:38:10.015 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2024-01-16T04:38:10.226 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2024-01-16T04:38:10.340 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2024-01-16T04:38:10.376 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2024-01-16T04:38:10.497 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2024-01-16T04:38:10.635 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2024-01-16T04:38:10.807 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2024-01-16T04:38:10.991 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2024-01-16T04:38:11.255 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2024-01-16T04:38:12.869 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2024-01-16T04:38:12.996 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2024-01-16T04:38:13.207 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2024-01-16T04:38:13.378 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2024-01-16T04:38:13.648 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2024-01-16T04:38:13.820 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2024-01-16T04:38:13.904 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2024-01-16T04:38:14.068 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2024-01-16T04:38:14.280 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2024-01-16T04:38:14.355 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2024-01-16T04:38:14.420 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2024-01-16T04:38:14.485 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2024-01-16T04:38:14.549 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2024-01-16T04:38:14.692 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2024-01-16T04:38:14.757 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2024-01-16T04:38:14.822 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2024-01-16T04:38:14.890 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2024-01-16T04:38:14.955 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2024-01-16T04:38:15.006 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2024-01-16T04:38:15.078 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2024-01-16T04:38:15.199 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2024-01-16T04:38:15.296 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2024-01-16T04:38:15.369 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2024-01-16T04:38:15.475 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2024-01-16T04:38:15.577 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2024-01-16T04:38:15.680 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2024-01-16T04:38:15.779 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2024-01-16T04:38:15.880 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2024-01-16T04:38:15.975 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2024-01-16T04:38:16.027 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2024-01-16T04:38:16.101 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2024-01-16T04:38:16.179 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2024-01-16T04:38:16.311 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2024-01-16T04:38:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2024-01-16T04:38:16.481 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2024-01-16T04:38:16.531 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2024-01-16T04:38:16.601 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2024-01-16T04:38:16.661 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2024-01-16T04:38:16.784 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifmatch_good ... ok 2024-01-16T04:38:16.908 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifnonematch_failed ... ok 2024-01-16T04:38:16.928 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_user_policy ... ok 2024-01-16T04:38:16.955 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2024-01-16T04:38:17.032 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2024-01-16T04:38:17.108 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2024-01-16T04:38:17.184 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2024-01-16T04:38:17.264 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2024-01-16T04:38:17.338 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2024-01-16T04:38:17.364 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2024-01-16T04:38:17.471 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2024-01-16T04:38:17.676 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2024-01-16T04:38:17.842 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2024-01-16T04:38:17.918 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2024-01-16T04:38:18.107 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2024-01-16T04:39:45.840 INFO:teuthology.orchestra.run.smithi038.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2024-01-16T04:39:45.840 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr:====================================================================== 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr:FAIL: s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr:---------------------------------------------------------------------- 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr:Traceback (most recent call last): 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr: self.test(*self.arg) 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 4439, in test_bucket_recreate_overwrite_acl 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr: e = assert_raises(ClientError, client.create_bucket, Bucket=bucket_name) 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/utils.py", line 21, in assert_raises 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr: raise AssertionError("%s not raised" % excName) 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr:AssertionError: ClientError not raised 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr:-------------------- >> begin captured logging << -------------------- 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2024-01-16T04:39:45.841 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f6242024598> 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:39:45.842 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-vlzrd5iggfgho3s-211', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-acl': 'public-read', 'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-532.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi038.front.sepia.ceph.com:80/test-client.0-vlzrd5iggfgho3s-211', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-vlzrd5iggfgho3s-211'}}} 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:PUT 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:/test-client.0-vlzrd5iggfgho3s-211 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:host:smithi038.front.sepia.ceph.com 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:x-amz-acl:public-read 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:39:45.843 INFO:teuthology.orchestra.run.smithi038.stderr:x-amz-date:20240116T043646Z 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: StringToSign: 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:AWS4-HMAC-SHA256 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:20240116T043646Z 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:20240116/us-east-1/s3/aws4_request 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:4bee88c34735bb6b900e638db729c0fcbdfca1425d09474868fb92a6efa30446 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: Signature: 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:0c29b489c446e052d4d9f22ffcab16da629acfc87a57e3f391ddff335d810a3a 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi038.front.sepia.ceph.com:80 2024-01-16T04:39:45.844 INFO:teuthology.orchestra.run.smithi038.stderr:urllib3.connectionpool: DEBUG: http://smithi038.front.sepia.ceph.com:80 "PUT /test-client.0-vlzrd5iggfgho3s-211 HTTP/1.1" 200 0 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000b28478b8e41454c7-0065a607de-1061-default', 'Content-Length': '0', 'Date': 'Tue, 16 Jan 2024 04:36:46 GMT'} 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.parsers: DEBUG: Response body: 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:b'' 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:39:45.845 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-vlzrd5iggfgho3s-211', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-532.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi038.front.sepia.ceph.com:80/test-client.0-vlzrd5iggfgho3s-211', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-vlzrd5iggfgho3s-211'}}} 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:PUT 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:/test-client.0-vlzrd5iggfgho3s-211 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:host:smithi038.front.sepia.ceph.com 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:x-amz-date:20240116T043646Z 2024-01-16T04:39:45.846 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:host;x-amz-content-sha256;x-amz-date 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: StringToSign: 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:AWS4-HMAC-SHA256 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:20240116T043646Z 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:20240116/us-east-1/s3/aws4_request 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:060fda229e77a5f31d336228850ee06f6b14ba00c1ae1ef0a44d4f5b52b9c059 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: Signature: 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:50390b217d139db36c6c54be4dc2ea10f2dfc221a8370ed9caeb448d75d2134b 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:urllib3.connectionpool: DEBUG: http://smithi038.front.sepia.ceph.com:80 "PUT /test-client.0-vlzrd5iggfgho3s-211 HTTP/1.1" 200 0 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000c967d9325c2cff2d-0065a607de-1061-default', 'Content-Length': '0', 'Date': 'Tue, 16 Jan 2024 04:36:46 GMT'} 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.parsers: DEBUG: Response body: 2024-01-16T04:39:45.847 INFO:teuthology.orchestra.run.smithi038.stderr:b'' 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr:--------------------- >> end captured logging << --------------------- 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr:====================================================================== 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr:FAIL: s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr:---------------------------------------------------------------------- 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr:Traceback (most recent call last): 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr: self.test(*self.arg) 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 4453, in test_bucket_recreate_new_acl 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr: e = assert_raises(ClientError, client.create_bucket, Bucket=bucket_name, ACL='public-read') 2024-01-16T04:39:45.848 INFO:teuthology.orchestra.run.smithi038.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/utils.py", line 21, in assert_raises 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr: raise AssertionError("%s not raised" % excName) 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:AssertionError: ClientError not raised 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:-------------------- >> begin captured logging << -------------------- 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f6242024598> 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:39:45.849 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-vlzrd5iggfgho3s-212', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-532.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi038.front.sepia.ceph.com:80/test-client.0-vlzrd5iggfgho3s-212', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-vlzrd5iggfgho3s-212'}}} 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:PUT 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:/test-client.0-vlzrd5iggfgho3s-212 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:45.850 INFO:teuthology.orchestra.run.smithi038.stderr:host:smithi038.front.sepia.ceph.com 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:x-amz-date:20240116T043646Z 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:host;x-amz-content-sha256;x-amz-date 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: StringToSign: 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:AWS4-HMAC-SHA256 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:20240116T043646Z 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:20240116/us-east-1/s3/aws4_request 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:829c70f02b8a156b2e05dcd4d3951adab4bfb60d82a103738135b285b001142d 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: Signature: 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:33c481286a451841a173d6219167a139c49e46d60cbed60945f5a446baf1d13a 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-01-16T04:39:45.851 INFO:teuthology.orchestra.run.smithi038.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi038.front.sepia.ceph.com:80 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:urllib3.connectionpool: DEBUG: http://smithi038.front.sepia.ceph.com:80 "PUT /test-client.0-vlzrd5iggfgho3s-212 HTTP/1.1" 200 0 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx000002dd3cf66a25c196e-0065a607de-1061-default', 'Content-Length': '0', 'Date': 'Tue, 16 Jan 2024 04:36:46 GMT'} 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.parsers: DEBUG: Response body: 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:b'' 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.852 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:39:45.853 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:39:45.853 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-vlzrd5iggfgho3s-212', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-acl': 'public-read', 'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-532.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi038.front.sepia.ceph.com:80/test-client.0-vlzrd5iggfgho3s-212', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-vlzrd5iggfgho3s-212'}}} 2024-01-16T04:39:45.853 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.853 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.853 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-01-16T04:39:45.853 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.853 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-01-16T04:39:45.853 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-01-16T04:39:45.853 INFO:teuthology.orchestra.run.smithi038.stderr:PUT 2024-01-16T04:39:45.853 INFO:teuthology.orchestra.run.smithi038.stderr:/test-client.0-vlzrd5iggfgho3s-212 2024-01-16T04:39:45.853 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:45.853 INFO:teuthology.orchestra.run.smithi038.stderr:host:smithi038.front.sepia.ceph.com 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:x-amz-acl:public-read 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:x-amz-date:20240116T043646Z 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: StringToSign: 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:AWS4-HMAC-SHA256 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:20240116T043646Z 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:20240116/us-east-1/s3/aws4_request 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:b59bc73622b81e2fea0c8cc0b9a425a2ec7a5bec8654e1816bf6a10674ea7b94 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.auth: DEBUG: Signature: 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:e3ea7f80970d9d66ba8c7775695037612a96cd513a450471f94ea2f22e12b3e8 2024-01-16T04:39:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr:urllib3.connectionpool: DEBUG: http://smithi038.front.sepia.ceph.com:80 "PUT /test-client.0-vlzrd5iggfgho3s-212 HTTP/1.1" 200 0 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000b80c220168eb62e8-0065a607de-1061-default', 'Content-Length': '0', 'Date': 'Tue, 16 Jan 2024 04:36:46 GMT'} 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.parsers: DEBUG: Response body: 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr:b'' 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr:--------------------- >> end captured logging << --------------------- 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr:---------------------------------------------------------------------- 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr:Ran 566 tests in 212.866s 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:45.855 INFO:teuthology.orchestra.run.smithi038.stderr:FAILED (SKIP=76, failures=2) 2024-01-16T04:39:45.912 INFO:teuthology.orchestra.run.smithi038.stderr:Exception ignored in: .generate at 0x7f62459c2a40> 2024-01-16T04:39:45.913 INFO:teuthology.orchestra.run.smithi038.stderr:RuntimeError: generator ignored GeneratorExit 2024-01-16T04:39:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:39:45.946 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 441, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi038 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2024-01-16T04:39:45.947 INFO:tasks.s3tests:Cleaning up boto... 2024-01-16T04:39:45.948 DEBUG:teuthology.orchestra.run.smithi038:> rm /home/ubuntu/cephtest/boto.cfg 2024-01-16T04:39:45.966 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2024-01-16T04:39:46.046 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:39:46.046 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:39:46.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.056+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.057+0000 7efd847017c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.057+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.057+0000 7efd847017c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.057+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.057+0000 7efd847017c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.057+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.058+0000 7efd847017c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.066+0000 7efd847017c0 20 RGWRados::pool_iterate: got zone_info.e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 2024-01-16T04:39:46.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.066+0000 7efd847017c0 20 RGWRados::pool_iterate: got zonegroup_info.78faec5d-171d-49c3-930a-c335c902808d 2024-01-16T04:39:46.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.066+0000 7efd847017c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:39:46.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.066+0000 7efd847017c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:39:46.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.066+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.066+0000 7efd847017c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.066+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.066+0000 7efd847017c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:39:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.066+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.066+0000 7efd847017c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.066+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.067+0000 7efd847017c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.067+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.067+0000 7efd847017c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.067+0000 7efd847017c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.067+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.068+0000 7efd847017c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.068+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.068+0000 7efd847017c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.068+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.068+0000 7efd847017c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.068+0000 7efd847017c0 20 zonegroup default 2024-01-16T04:39:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.068+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.068+0000 7efd847017c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.068+0000 7efd847017c0 10 Cannot find current period zone using local zone 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.068+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.068+0000 7efd847017c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.068+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.069+0000 7efd847017c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.069+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.069+0000 7efd847017c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.069+0000 7efd847017c0 20 zone default found 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.069+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.069+0000 7efd847017c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.069+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.069+0000 7efd847017c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.069+0000 7efd847017c0 20 started sync module instance, tier type = 2024-01-16T04:39:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.069+0000 7efd847017c0 20 started zone id=e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 (name=default) with tier type = 2024-01-16T04:39:46.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.084+0000 7efd847017c0 20 add_watcher() i=0 2024-01-16T04:39:46.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.084+0000 7efd847017c0 20 add_watcher() i=1 2024-01-16T04:39:46.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.084+0000 7efd847017c0 20 add_watcher() i=2 2024-01-16T04:39:46.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.084+0000 7efd847017c0 20 add_watcher() i=3 2024-01-16T04:39:46.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.084+0000 7efd847017c0 20 add_watcher() i=4 2024-01-16T04:39:46.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.084+0000 7efd847017c0 20 add_watcher() i=5 2024-01-16T04:39:46.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.084+0000 7efd847017c0 20 add_watcher() i=6 2024-01-16T04:39:46.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.085+0000 7efd847017c0 20 add_watcher() i=7 2024-01-16T04:39:46.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.085+0000 7efd847017c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:39:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.087+0000 7efd417e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:39:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.087+0000 7efd847017c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:39:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.087+0000 7efd847017c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:39:46.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.128+0000 7efcaffff700 20 reqs_thread_entry: start 2024-01-16T04:39:46.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.128+0000 7efcadffb700 20 reqs_thread_entry: start 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.128+0000 7efd847017c0 20 init_complete bucket index max shards: 11 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.129+0000 7efd847017c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.129+0000 7efc9bfff700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:40:16 2024 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.129+0000 7efc9b7fe700 20 reqs_thread_entry: start 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.129+0000 7efd847017c0 20 get_system_obj_state: rctx=0x7fffbd63d1d8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x5563cdc26d50 s->prefetch_data=0 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.129+0000 7efd847017c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.129+0000 7efd847017c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.129+0000 7efd847017c0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.129+0000 7efd847017c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.129+0000 7efd847017c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.129+0000 7efd847017c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.129+0000 7efd847017c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.130+0000 7efd847017c0 20 rados_obj.operate() r=0 bl.length=469 2024-01-16T04:39:46.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.130+0000 7efd847017c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x11 2024-01-16T04:39:46.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.130+0000 7efd847017c0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:39:46.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.130+0000 7efd847017c0 10 removing key index: TVYAECPWOXPTEUUVKKVE 2024-01-16T04:39:46.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.130+0000 7efd847017c0 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.keys:TVYAECPWOXPTEUUVKKVE, ofs0, ns] 2024-01-16T04:39:46.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.131+0000 7efd6d85d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824731 cookie 93887143527664 notifier 4235 bl.length()=170 2024-01-16T04:39:46.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.131+0000 7efd847017c0 10 removing email index: foo.client.0+test@test.test 2024-01-16T04:39:46.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.131+0000 7efd847017c0 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2024-01-16T04:39:46.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.131+0000 7efd6d05c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823687 cookie 93887143532544 notifier 4235 bl.length()=178 2024-01-16T04:39:46.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.133+0000 7efd847017c0 10 removing user buckets index 2024-01-16T04:39:46.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.133+0000 7efd847017c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2024-01-16T04:39:46.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.133+0000 7efd6d85d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823688 cookie 93887143513904 notifier 4235 bl.length()=169 2024-01-16T04:39:46.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.135+0000 7efd847017c0 10 removing user index: foo.client.0 2024-01-16T04:39:46.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.135+0000 7efd847017c0 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2024-01-16T04:39:46.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.135+0000 7efd847017c0 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-01-16T04:39:46.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.135+0000 7efd6d05c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824732 cookie 93887143503968 notifier 4235 bl.length()=161 2024-01-16T04:39:46.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.138+0000 7efd847017c0 20 remove_watcher() i=0 2024-01-16T04:39:46.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.138+0000 7efd847017c0 2 removed watcher, disabling cache 2024-01-16T04:39:46.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.139+0000 7efd847017c0 20 remove_watcher() i=1 2024-01-16T04:39:46.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.140+0000 7efd847017c0 20 remove_watcher() i=2 2024-01-16T04:39:46.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.141+0000 7efd847017c0 20 remove_watcher() i=3 2024-01-16T04:39:46.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.142+0000 7efd847017c0 20 remove_watcher() i=4 2024-01-16T04:39:46.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.143+0000 7efd847017c0 20 remove_watcher() i=5 2024-01-16T04:39:46.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.143+0000 7efd847017c0 20 remove_watcher() i=6 2024-01-16T04:39:46.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.144+0000 7efd847017c0 20 remove_watcher() i=7 2024-01-16T04:39:46.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.145+0000 7efd417e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:39:46.147 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2024-01-16T04:39:46.229 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:39:46.230 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:39:46.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.238+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.239+0000 7f8d9cecc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.239+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.239+0000 7f8d9cecc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.239+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.239+0000 7f8d9cecc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.239+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.239+0000 7f8d9cecc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.247+0000 7f8d9cecc7c0 20 RGWRados::pool_iterate: got zone_info.e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 2024-01-16T04:39:46.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.247+0000 7f8d9cecc7c0 20 RGWRados::pool_iterate: got zonegroup_info.78faec5d-171d-49c3-930a-c335c902808d 2024-01-16T04:39:46.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.247+0000 7f8d9cecc7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:39:46.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.247+0000 7f8d9cecc7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:39:46.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.247+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.247+0000 7f8d9cecc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.247+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.247+0000 7f8d9cecc7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:39:46.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.247+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.247+0000 7f8d9cecc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.247+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.248+0000 7f8d9cecc7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:39:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.248+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.248+0000 7f8d9cecc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.248+0000 7f8d9cecc7c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:39:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.248+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.248+0000 7f8d9cecc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.248+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.248+0000 7f8d9cecc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.248+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.248+0000 7f8d9cecc7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:39:46.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.249+0000 7f8d9cecc7c0 20 zonegroup default 2024-01-16T04:39:46.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.249+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.249+0000 7f8d9cecc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.249+0000 7f8d9cecc7c0 10 Cannot find current period zone using local zone 2024-01-16T04:39:46.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.249+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.249+0000 7f8d9cecc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.249+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.249+0000 7f8d9cecc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.249+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.249+0000 7f8d9cecc7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:39:46.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.249+0000 7f8d9cecc7c0 20 zone default found 2024-01-16T04:39:46.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.249+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.250+0000 7f8d9cecc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.250+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.250+0000 7f8d9cecc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.250+0000 7f8d9cecc7c0 20 started sync module instance, tier type = 2024-01-16T04:39:46.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.250+0000 7f8d9cecc7c0 20 started zone id=e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 (name=default) with tier type = 2024-01-16T04:39:46.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.262+0000 7f8d9cecc7c0 20 add_watcher() i=0 2024-01-16T04:39:46.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.262+0000 7f8d9cecc7c0 20 add_watcher() i=1 2024-01-16T04:39:46.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.262+0000 7f8d9cecc7c0 20 add_watcher() i=2 2024-01-16T04:39:46.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.262+0000 7f8d9cecc7c0 20 add_watcher() i=3 2024-01-16T04:39:46.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.262+0000 7f8d9cecc7c0 20 add_watcher() i=4 2024-01-16T04:39:46.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.262+0000 7f8d9cecc7c0 20 add_watcher() i=5 2024-01-16T04:39:46.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.262+0000 7f8d9cecc7c0 20 add_watcher() i=6 2024-01-16T04:39:46.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.263+0000 7f8d9cecc7c0 20 add_watcher() i=7 2024-01-16T04:39:46.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.263+0000 7f8d9cecc7c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:39:46.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.264+0000 7f8d59fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:39:46.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.264+0000 7f8d9cecc7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:39:46.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.264+0000 7f8d9cecc7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:39:46.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.307+0000 7f8cd0ff9700 20 reqs_thread_entry: start 2024-01-16T04:39:46.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8cc67fc700 20 reqs_thread_entry: start 2024-01-16T04:39:46.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8d9cecc7c0 20 init_complete bucket index max shards: 11 2024-01-16T04:39:46.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8d9cecc7c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:39:46.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8cc4ff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:40:16 2024 2024-01-16T04:39:46.307 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:46.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8cabfff700 20 reqs_thread_entry: start 2024-01-16T04:39:46.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8d9cecc7c0 20 get_system_obj_state: rctx=0x7fff05fb03c8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x563755227d50 s->prefetch_data=0 2024-01-16T04:39:46.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8d9cecc7c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-01-16T04:39:46.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8d9cecc7c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2024-01-16T04:39:46.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8d9cecc7c0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:39:46.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8d9cecc7c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:39:46.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8d9cecc7c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:39:46.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8d9cecc7c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:39:46.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.308+0000 7f8d9cecc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.309+0000 7f8d9cecc7c0 20 rados_obj.operate() r=0 bl.length=469 2024-01-16T04:39:46.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.309+0000 7f8d9cecc7c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x11 2024-01-16T04:39:46.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.309+0000 7f8d9cecc7c0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:39:46.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.309+0000 7f8d9cecc7c0 10 removing key index: EUTPDQQLUHNECISCHSYU 2024-01-16T04:39:46.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.309+0000 7f8d9cecc7c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:EUTPDQQLUHNECISCHSYU, ofs0, ns] 2024-01-16T04:39:46.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.309+0000 7f8d86028700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823689 cookie 94795652833120 notifier 4241 bl.length()=170 2024-01-16T04:39:46.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.311+0000 7f8d9cecc7c0 10 removing email index: bar.client.0+test@test.test 2024-01-16T04:39:46.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.311+0000 7f8d9cecc7c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2024-01-16T04:39:46.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.311+0000 7f8d86028700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823690 cookie 94795652833120 notifier 4241 bl.length()=178 2024-01-16T04:39:46.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.312+0000 7f8d9cecc7c0 10 removing user buckets index 2024-01-16T04:39:46.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.312+0000 7f8d9cecc7c0 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2024-01-16T04:39:46.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.313+0000 7f8d86028700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618822702 cookie 94795652837232 notifier 4241 bl.length()=169 2024-01-16T04:39:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.314+0000 7f8d9cecc7c0 10 removing user index: bar.client.0 2024-01-16T04:39:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.314+0000 7f8d9cecc7c0 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2024-01-16T04:39:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.314+0000 7f8d9cecc7c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-01-16T04:39:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.314+0000 7f8d86028700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823691 cookie 94795652833120 notifier 4241 bl.length()=161 2024-01-16T04:39:46.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.317+0000 7f8d9cecc7c0 20 remove_watcher() i=0 2024-01-16T04:39:46.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.317+0000 7f8d9cecc7c0 2 removed watcher, disabling cache 2024-01-16T04:39:46.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.318+0000 7f8d9cecc7c0 20 remove_watcher() i=1 2024-01-16T04:39:46.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.319+0000 7f8d9cecc7c0 20 remove_watcher() i=2 2024-01-16T04:39:46.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.320+0000 7f8d9cecc7c0 20 remove_watcher() i=3 2024-01-16T04:39:46.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.320+0000 7f8d9cecc7c0 20 remove_watcher() i=4 2024-01-16T04:39:46.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.321+0000 7f8d9cecc7c0 20 remove_watcher() i=5 2024-01-16T04:39:46.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.322+0000 7f8d9cecc7c0 20 remove_watcher() i=6 2024-01-16T04:39:46.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.323+0000 7f8d9cecc7c0 20 remove_watcher() i=7 2024-01-16T04:39:46.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.323+0000 7f8d59fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:39:46.326 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2024-01-16T04:39:46.407 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:39:46.407 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:39:46.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.415+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.416+0000 7fd17aec07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.416+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.416+0000 7fd17aec07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.416+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.416+0000 7fd17aec07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.416+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.417+0000 7fd17aec07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.424+0000 7fd17aec07c0 20 RGWRados::pool_iterate: got zone_info.e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 2024-01-16T04:39:46.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.424+0000 7fd17aec07c0 20 RGWRados::pool_iterate: got zonegroup_info.78faec5d-171d-49c3-930a-c335c902808d 2024-01-16T04:39:46.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.424+0000 7fd17aec07c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:39:46.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.424+0000 7fd17aec07c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:39:46.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.424+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.424+0000 7fd17aec07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.424+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.424+0000 7fd17aec07c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:39:46.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.424+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.425+0000 7fd17aec07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.425+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.425+0000 7fd17aec07c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:39:46.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.425+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.425+0000 7fd17aec07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.425+0000 7fd17aec07c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:39:46.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.425+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.425+0000 7fd17aec07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.425+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.425+0000 7fd17aec07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.425+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.426+0000 7fd17aec07c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.426+0000 7fd17aec07c0 20 zonegroup default 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.426+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.426+0000 7fd17aec07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.426+0000 7fd17aec07c0 10 Cannot find current period zone using local zone 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.426+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.426+0000 7fd17aec07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.426+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.426+0000 7fd17aec07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.426+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.427+0000 7fd17aec07c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.427+0000 7fd17aec07c0 20 zone default found 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.427+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.427+0000 7fd17aec07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.427+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.427+0000 7fd17aec07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:39:46.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.427+0000 7fd17aec07c0 20 started sync module instance, tier type = 2024-01-16T04:39:46.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.427+0000 7fd17aec07c0 20 started zone id=e6dbdbee-2e9f-42f0-a97a-ac241c8e34a2 (name=default) with tier type = 2024-01-16T04:39:46.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.448+0000 7fd17aec07c0 20 add_watcher() i=0 2024-01-16T04:39:46.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.448+0000 7fd17aec07c0 20 add_watcher() i=1 2024-01-16T04:39:46.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.448+0000 7fd17aec07c0 20 add_watcher() i=2 2024-01-16T04:39:46.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.448+0000 7fd17aec07c0 20 add_watcher() i=3 2024-01-16T04:39:46.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.448+0000 7fd17aec07c0 20 add_watcher() i=4 2024-01-16T04:39:46.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.448+0000 7fd17aec07c0 20 add_watcher() i=5 2024-01-16T04:39:46.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.448+0000 7fd17aec07c0 20 add_watcher() i=6 2024-01-16T04:39:46.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.449+0000 7fd17aec07c0 20 add_watcher() i=7 2024-01-16T04:39:46.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.449+0000 7fd17aec07c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:39:46.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.451+0000 7fd137fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:39:46.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.451+0000 7fd17aec07c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:39:46.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.451+0000 7fd17aec07c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:39:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.492+0000 7fd0a6ffd700 20 reqs_thread_entry: start 2024-01-16T04:39:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.492+0000 7fd0a4ff9700 20 reqs_thread_entry: start 2024-01-16T04:39:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd17aec07c0 20 init_complete bucket index max shards: 11 2024-01-16T04:39:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd17aec07c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:39:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd09affd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:40:16 2024 2024-01-16T04:39:46.492 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:39:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd09a7fc700 20 reqs_thread_entry: start 2024-01-16T04:39:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd17aec07c0 20 get_system_obj_state: rctx=0x7ffdca07dbf8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x558a65e23d50 s->prefetch_data=0 2024-01-16T04:39:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd17aec07c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-01-16T04:39:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd17aec07c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2024-01-16T04:39:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd17aec07c0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:39:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd17aec07c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:39:46.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd17aec07c0 10 updating xattr: name=user.rgw.user-policy bl.length()=118 2024-01-16T04:39:46.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd17aec07c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:39:46.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd17aec07c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:39:46.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.493+0000 7fd17aec07c0 20 rados->read ofs=0 len=0 2024-01-16T04:39:46.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.494+0000 7fd17aec07c0 20 rados_obj.operate() r=0 bl.length=528 2024-01-16T04:39:46.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.494+0000 7fd17aec07c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x11 2024-01-16T04:39:46.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.494+0000 7fd17aec07c0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:39:46.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.494+0000 7fd17aec07c0 10 removing key index: LMHZFGXTIMPAKJZGYPFN 2024-01-16T04:39:46.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.494+0000 7fd17aec07c0 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:LMHZFGXTIMPAKJZGYPFN, ofs0, ns] 2024-01-16T04:39:46.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.494+0000 7fd157fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618822703 cookie 94052904494800 notifier 4247 bl.length()=170 2024-01-16T04:39:46.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.495+0000 7fd17aec07c0 10 removing email index: testx$tenanteduser.client.0+test@test.test 2024-01-16T04:39:46.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.495+0000 7fd17aec07c0 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2024-01-16T04:39:46.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.496+0000 7fd1577fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824733 cookie 94052904512400 notifier 4247 bl.length()=193 2024-01-16T04:39:46.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.497+0000 7fd17aec07c0 10 removing user buckets index 2024-01-16T04:39:46.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.497+0000 7fd17aec07c0 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2024-01-16T04:39:46.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.497+0000 7fd157fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823692 cookie 94052904505520 notifier 4247 bl.length()=184 2024-01-16T04:39:46.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.498+0000 7fd17aec07c0 10 removing user index: testx$tenanteduser.client.0 2024-01-16T04:39:46.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.498+0000 7fd17aec07c0 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2024-01-16T04:39:46.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.498+0000 7fd17aec07c0 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-01-16T04:39:46.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.499+0000 7fd1577fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824734 cookie 94052904495904 notifier 4247 bl.length()=176 2024-01-16T04:39:46.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.502+0000 7fd17aec07c0 20 remove_watcher() i=0 2024-01-16T04:39:46.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.502+0000 7fd17aec07c0 2 removed watcher, disabling cache 2024-01-16T04:39:46.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.503+0000 7fd17aec07c0 20 remove_watcher() i=1 2024-01-16T04:39:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.504+0000 7fd17aec07c0 20 remove_watcher() i=2 2024-01-16T04:39:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.505+0000 7fd17aec07c0 20 remove_watcher() i=3 2024-01-16T04:39:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.506+0000 7fd17aec07c0 20 remove_watcher() i=4 2024-01-16T04:39:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.506+0000 7fd17aec07c0 20 remove_watcher() i=5 2024-01-16T04:39:46.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.507+0000 7fd17aec07c0 20 remove_watcher() i=6 2024-01-16T04:39:46.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.508+0000 7fd17aec07c0 20 remove_watcher() i=7 2024-01-16T04:39:46.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-01-16T04:39:46.508+0000 7fd137fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:39:46.511 INFO:tasks.s3tests:Removing s3-tests... 2024-01-16T04:39:46.511 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/s3-tests 2024-01-16T04:39:46.675 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 639, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 59, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 441, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi038 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2024-01-16T04:39:46.906 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ecfc007a35be48e89baee529444c1189 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 639, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 59, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 441, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi038 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2024-01-16T04:39:46.909 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2024-01-16T04:39:46.919 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-01-16T04:39:46.963 DEBUG:tasks.rgw.client.0:waiting for process to exit 2024-01-16T04:39:46.964 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:39:46.968 INFO:tasks.rgw.client.0.smithi038.stdout:2024-01-16T04:39:46.967+0000 7ff610841700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends civetweb port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground (PID: 42983) UID: 0 2024-01-16T04:39:46.969 INFO:tasks.rgw.client.0.smithi038.stdout:2024-01-16T04:39:46.967+0000 7ff62853d880 -1 shutting down 2024-01-16T04:39:50.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2024-01-16T04:39:53.065 INFO:tasks.rgw.client.0:Stopped 2024-01-16T04:39:53.066 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2024-01-16T04:39:53.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/vault-root-token 2024-01-16T04:39:53.166 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-16T04:39:53.180 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-01-16T04:39:53.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T04:39:53.516 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-01-16T04:39:53.517 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":36,"fsid":"752b6117-c411-45ed-8b00-1e36318c8108","created":"2024-01-16T04:34:53.569874+0000","modified":"2024-01-16T04:37:01.564553+0000","last_up_change":"2024-01-16T04:34:59.963811+0000","last_in_change":"2024-01-16T04:34:54.517362+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-16T04:35:00.023083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-16T04:35:02.294714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2024-01-16T04:35:09.045248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2024-01-16T04:35:10.941395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2024-01-16T04:35:12.817342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2024-01-16T04:35:14.681054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2024-01-16T04:35:16.666231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2024-01-16T04:35:18.717869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2024-01-16T04:35:28.800055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2024-01-16T04:36:14.166933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"4244da9f-308a-4ce5-9b56-a190f438a602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":40407},{"type":"v1","addr":"172.21.15.38:6801","nonce":40407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":40407},{"type":"v1","addr":"172.21.15.38:6803","nonce":40407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":40407},{"type":"v1","addr":"172.21.15.38:6807","nonce":40407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":40407},{"type":"v1","addr":"172.21.15.38:6805","nonce":40407}]},"public_addr":"172.21.15.38:6801/40407","cluster_addr":"172.21.15.38:6803/40407","heartbeat_back_addr":"172.21.15.38:6807/40407","heartbeat_front_addr":"172.21.15.38:6805/40407","state":["exists","up"]},{"osd":1,"uuid":"f6b91531-55fd-49cf-a96f-57dc23c4bd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":40408},{"type":"v1","addr":"172.21.15.38:6809","nonce":40408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":40408},{"type":"v1","addr":"172.21.15.38:6811","nonce":40408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":40408},{"type":"v1","addr":"172.21.15.38:6815","nonce":40408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":40408},{"type":"v1","addr":"172.21.15.38:6813","nonce":40408}]},"public_addr":"172.21.15.38:6809/40408","cluster_addr":"172.21.15.38:6811/40408","heartbeat_back_addr":"172.21.15.38:6815/40408","heartbeat_front_addr":"172.21.15.38:6813/40408","state":["exists","up"]},{"osd":2,"uuid":"ee72b7e9-cfcb-4142-ac3d-4d1e52bbb4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":40437},{"type":"v1","addr":"172.21.15.38:6817","nonce":40437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":40437},{"type":"v1","addr":"172.21.15.38:6819","nonce":40437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":40437},{"type":"v1","addr":"172.21.15.38:6823","nonce":40437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":40437},{"type":"v1","addr":"172.21.15.38:6821","nonce":40437}]},"public_addr":"172.21.15.38:6817/40437","cluster_addr":"172.21.15.38:6819/40437","heartbeat_back_addr":"172.21.15.38:6823/40437","heartbeat_front_addr":"172.21.15.38:6821/40437","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:34:58.931224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:34:58.897675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:34:58.911909+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":0,"to":2}]},{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":0}]},{"pgid":"4.20","mappings":[{"from":1,"to":0}]},{"pgid":"4.2a","mappings":[{"from":1,"to":2}]},{"pgid":"4.30","mappings":[{"from":1,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":2}]},{"pgid":"10.10","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:39:53.530 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-16T04:39:53.848 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-16T04:39:53.849 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-16T04:39:53.849 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-16T04:39:53.849 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:39:53.895 INFO:tasks.ceph.osd.0:Stopped 2024-01-16T04:39:53.896 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-16T04:39:53.896 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:39:53.945 INFO:tasks.ceph.osd.1:Stopped 2024-01-16T04:39:53.945 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-16T04:39:53.946 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:39:53.987 INFO:tasks.ceph.osd.2:Stopped 2024-01-16T04:39:53.987 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-16T04:39:53.988 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-16T04:39:53.988 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:39:54.008 INFO:tasks.ceph.mgr.x:Stopped 2024-01-16T04:39:54.009 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-16T04:39:54.009 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-16T04:39:54.009 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:39:54.059 INFO:tasks.ceph.mon.a:Stopped 2024-01-16T04:39:54.060 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-16T04:39:54.060 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-01-16T04:39:54.129 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi038.front.sepia.ceph.com 2024-01-16T04:39:54.129 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-16T04:39:54.724 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi038.front.sepia.ceph.com 2024-01-16T04:39:54.724 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-16T04:39:54.823 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi038.front.sepia.ceph.com 2024-01-16T04:39:54.824 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-16T04:39:54.893 INFO:tasks.ceph:Archiving mon data... 2024-01-16T04:39:54.895 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517318/data/mon.a.tgz 2024-01-16T04:39:54.895 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-01-16T04:39:54.914 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.lDxpbvEVLY 2024-01-16T04:39:54.914 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.lDxpbvEVLY 2024-01-16T04:39:55.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2024-01-16T04:39:55.218 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.lDxpbvEVLY 2024-01-16T04:39:55.306 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.lDxpbvEVLY is 998KB 2024-01-16T04:39:55.412 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.lDxpbvEVLY 2024-01-16T04:39:55.429 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-16T04:39:55.429 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-16T04:39:55.496 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-01-16T04:39:55.496 DEBUG:teuthology.orchestra.run.smithi038:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-01-16T04:39:55.565 INFO:teuthology.orchestra.run.smithi038.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-01-16T04:39:55.571 INFO:tasks.ceph:Archiving crash dumps... 2024-01-16T04:39:55.572 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517318/remote/smithi038/crash 2024-01-16T04:39:55.573 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-16T04:39:55.638 INFO:tasks.ceph:Compressing logs... 2024-01-16T04:39:55.638 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T04:39:55.713 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42718.log 2024-01-16T04:39:55.713 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44226.log 2024-01-16T04:39:55.713 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42081.log 2024-01-16T04:39:55.714 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42718.log.gz 2024-01-16T04:39:55.714 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.0.44226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42375.log 2024-01-16T04:39:55.714 INFO:teuthology.orchestra.run.smithi038.stderr: 83.4% -- replaced with /var/log/ceph/ceph-client.0.44226.log.gz 2024-01-16T04:39:55.714 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43734.log 2024-01-16T04:39:55.714 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42081.log.gz 2024-01-16T04:39:55.714 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42522.log 2024-01-16T04:39:55.714 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42375.log.gz 2024-01-16T04:39:55.714 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.0.43734.log: gzip -5 --verbose -- /var/log/ceph/rgw.ceph.client.0.log 2024-01-16T04:39:55.715 INFO:teuthology.orchestra.run.smithi038.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.0.43734.log.gz 2024-01-16T04:39:55.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42522.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-16T04:39:55.715 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42522.log.gz 2024-01-16T04:39:55.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/rgw.ceph.client.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2024-01-16T04:39:55.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40225.log 2024-01-16T04:39:55.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42228.log 2024-01-16T04:39:55.715 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2024-01-16T04:39:55.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42861.log 2024-01-16T04:39:55.716 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40225.log.gz 2024-01-16T04:39:55.716 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42326.log 2024-01-16T04:39:55.716 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42228.log.gz 2024-01-16T04:39:55.716 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42910.log 2024-01-16T04:39:55.716 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42861.log.gz 2024-01-16T04:39:55.716 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42326.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-16T04:39:55.716 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42326.log.gz 2024-01-16T04:39:55.716 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44128.log 2024-01-16T04:39:55.716 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42910.log.gz 2024-01-16T04:39:55.717 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42620.log 2024-01-16T04:39:55.717 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44128.log: /var/log/ceph/ceph-client.0.43932.log 2024-01-16T04:39:55.717 INFO:teuthology.orchestra.run.smithi038.stderr: 88.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-16T04:39:55.717 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42179.log 2024-01-16T04:39:55.717 INFO:teuthology.orchestra.run.smithi038.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.0.44128.log.gz 2024-01-16T04:39:55.717 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42620.log.gz 2024-01-16T04:39:55.717 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.0.43932.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-16T04:39:55.717 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46601.log 2024-01-16T04:39:55.717 INFO:teuthology.orchestra.run.smithi038.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.0.43932.log.gz 2024-01-16T04:39:55.718 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42179.log.gz 2024-01-16T04:39:55.718 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46554.log 2024-01-16T04:39:55.718 INFO:teuthology.orchestra.run.smithi038.stderr:gzip /var/log/ceph/ceph-client.admin.46601.log: -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-16T04:39:55.718 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46601.log.gz 2024-01-16T04:39:55.718 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40272.log 2024-01-16T04:39:55.718 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46554.log.gz 2024-01-16T04:39:55.718 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46163.log 2024-01-16T04:39:55.718 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40272.log.gz 2024-01-16T04:39:55.719 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.0.46163.log: 83.3% -- replaced with /var/log/ceph/ceph-client.0.46163.log.gz 2024-01-16T04:39:55.719 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41963.log 2024-01-16T04:39:55.719 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-16T04:39:55.719 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43832.log 2024-01-16T04:39:55.720 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41963.log.gz 2024-01-16T04:39:55.721 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46261.log 2024-01-16T04:39:55.721 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.0.43832.log: 83.1% -- replaced with /var/log/ceph/ceph-client.0.43832.log.gz 2024-01-16T04:39:55.722 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.0.46261.log: 83.4% -- replaced with /var/log/ceph/ceph-client.0.46261.log.gz 2024-01-16T04:39:55.722 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42814.log 2024-01-16T04:39:55.722 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42032.log 2024-01-16T04:39:55.722 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42814.log.gz 2024-01-16T04:39:55.722 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42032.log.gz 2024-01-16T04:39:55.722 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-16T04:39:55.722 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42277.log 2024-01-16T04:39:55.722 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.42277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42277.log.gz 2024-01-16T04:39:55.723 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.36467.log 2024-01-16T04:39:55.723 INFO:teuthology.orchestra.run.smithi038.stderr: 86.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-16T04:39:55.723 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.tmp-client.admin.36467.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.36467.log.gz 2024-01-16T04:39:55.723 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42473.log 2024-01-16T04:39:55.723 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42130.log 2024-01-16T04:39:55.723 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42473.log.gz 2024-01-16T04:39:55.723 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42130.log.gz 2024-01-16T04:39:55.723 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42424.log 2024-01-16T04:39:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44030.log 2024-01-16T04:39:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42424.log.gz 2024-01-16T04:39:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.0.44030.log: 83.2% -- replaced with /var/log/ceph/ceph-client.0.44030.log.gz 2024-01-16T04:39:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46359.log 2024-01-16T04:39:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42571.log 2024-01-16T04:39:55.725 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.0.46359.log: /var/log/ceph/ceph-client.admin.42571.log: 83.4% -- replaced with /var/log/ceph/ceph-client.0.46359.log.gz 2024-01-16T04:39:55.726 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42571.log.gz 2024-01-16T04:39:55.726 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39990.log 2024-01-16T04:39:55.726 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42669.log 2024-01-16T04:39:55.726 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39990.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39990.log.gz 2024-01-16T04:39:55.726 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42669.log.gz 2024-01-16T04:39:55.726 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-16T04:39:55.726 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40176.log 2024-01-16T04:39:55.731 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.40176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40176.log.gz 2024-01-16T04:39:55.731 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42765.log 2024-01-16T04:39:55.731 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42765.log.gz 2024-01-16T04:39:55.732 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40406.log 2024-01-16T04:39:55.732 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40406.log.gz 2024-01-16T04:39:55.738 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-16T04:39:55.846 INFO:teuthology.orchestra.run.smithi038.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-16T04:39:56.880 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/rgw.ceph.client.0.log.gz 2024-01-16T04:40:00.792 INFO:teuthology.orchestra.run.smithi038.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-16T04:40:02.411 INFO:teuthology.orchestra.run.smithi038.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-16T04:40:02.758 INFO:teuthology.orchestra.run.smithi038.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-16T04:40:02.760 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-01-16T04:40:02.760 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m7.063s 2024-01-16T04:40:02.760 INFO:teuthology.orchestra.run.smithi038.stderr:user 0m19.486s 2024-01-16T04:40:02.760 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m0.685s 2024-01-16T04:40:02.760 INFO:tasks.ceph:Archiving logs... 2024-01-16T04:40:02.761 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517318/remote/smithi038/log 2024-01-16T04:40:02.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T04:40:05.517 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-16T04:40:05.526 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-16T04:40:05.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T04:40:05.599 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-16T04:40:05.607 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-16T04:40:05.608 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T04:40:05.632 INFO:teuthology.orchestra.run.smithi038.stderr:bash: ntpq: command not found 2024-01-16T04:40:05.636 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T04:40:05.636 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-01-16T04:40:05.636 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +2519us[+2519us] +/- 81ms 2024-01-16T04:40:05.637 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv02.front.sepia.ceph.com 3 6 377 23 +69us[ +64us] +/- 65ms 2024-01-16T04:40:05.637 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -2649us[-2649us] +/- 83ms 2024-01-16T04:40:05.637 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:40:05.638 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-16T04:40:05.646 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-16T04:40:05.647 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-16T04:40:05.676 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T04:40:05.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T04:40:05.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:40:05.777 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-16T04:40:05.787 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-16T04:40:05.809 INFO:teuthology.task.internal:Duration was 904.695466 seconds 2024-01-16T04:40:05.810 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-16T04:40:05.843 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-16T04:40:05.843 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T04:40:05.892 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T04:40:06.084 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-16T04:40:06.084 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-01-16T04:40:06.084 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T04:40:06.108 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-16T04:40:06.108 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T04:40:06.191 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-16T04:40:06.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T04:40:06.316 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-16T04:40:06.326 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-16T04:40:06.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T04:40:06.355 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-16T04:40:06.368 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T04:40:06.422 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-01-16T04:40:06.440 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T04:40:06.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:40:06.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-16T04:40:06.507 INFO:teuthology.task.internal:Transferring archived files... 2024-01-16T04:40:06.508 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517318/remote/smithi038 2024-01-16T04:40:06.508 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T04:40:06.576 INFO:teuthology.task.internal:Removing archive directory... 2024-01-16T04:40:06.577 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T04:40:06.628 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-16T04:40:06.639 INFO:teuthology.task.internal:Not uploading archives. 2024-01-16T04:40:06.640 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-16T04:40:06.661 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-16T04:40:06.661 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T04:40:06.686 INFO:teuthology.orchestra.run.smithi038.stdout: 525414 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 04:40 /home/ubuntu/cephtest 2024-01-16T04:40:06.688 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-16T04:40:06.700 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-16T04:40:06.758 INFO:teuthology.nuke:Checking targets against current locks 2024-01-16T04:40:06.784 DEBUG:teuthology.nuke:shortname: smithi038 2024-01-16T04:40:06.784 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T04:40:06.809 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517318', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 04:16:48.021793', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMrCOmXvvC5QExnYAs37aTRoqTzxJQMTjdrxQAm9itOC+nKbW1cyOatQX/3ZlUqvyvCKNjuPlA+30FlZFkyfV9k='} 2024-01-16T04:40:06.861 INFO:teuthology.orchestra.console.smithi038:Power off 2024-01-16T04:40:06.861 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T04:40:06.884 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off 2024-01-16T04:40:06.885 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T04:40:06.908 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-01-16T04:40:10.909 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T04:40:11.018 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-01-16T04:40:15.022 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T04:40:15.132 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-01-16T04:40:19.136 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T04:40:19.245 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off 2024-01-16T04:40:19.345 INFO:teuthology.orchestra.console.smithi038:Power off completed 2024-01-16T04:40:19.450 INFO:teuthology.run:Summary data: description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} duration: 904.6954655647278 failure_reason: 'Command failed (s3 tests against rgw) on smithi038 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a ''!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain''"' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ecfc007a35be48e89baee529444c1189 status: fail success: false 2024-01-16T04:40:19.451 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T04:40:19.534 INFO:teuthology.run:FAIL