2024-01-16T04:23:45.031 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-16T04:23:45.031 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517322 --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517322/orig.config.yaml 2024-01-16T04:23:45.061 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T04:23:45.135 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517322 branch: wip-yuri5-testing-2024-01-11-1300-pacific description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} email: yweinste@redhat.com first_in_suite: false job_id: '7517322' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri5-testing-2024-01-11-1300-pacific ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: vault rgw crypt vault auth: token rgw crypt vault prefix: /v1/kv/data rgw crypt vault secret engine: kv setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: default sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f rgw: client.0: use-vault-role: client.0 workunit: branch: wip-yuri5-testing-2024-01-11-1300-pacific sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9013 sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri5-testing-2024-01-11-1300-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f targets: smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMIJErr9dGrhWCSEySf7+DUoB2WqQZ+JCMm6HItnXIkSTf0om2vY9E3f8+9olOrKYtE9tuLk7Llog4RR4LbdBc= tasks: - install: null - ceph: null - vault: client.0: engine: kv install_sha256: 7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 install_url: https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip prefix: /v1/kv/data/ root_token: test_root_token secrets: - path: my-key-1 secret: a2V5MS5GcWVxKzhzTGNLaGtzQkg5NGVpb1FKcFpGb2c= - path: my-key-2 secret: a2V5Mi5yNUNNMGFzMVdIUVZxcCt5NGVmVGlQQ1k4YWg= - rgw: client.0: null - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 force-branch: ceph-pacific vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-15_22:05:22 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-16T04:23:45.135 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa; will attempt to use it 2024-01-16T04:23:45.136 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks 2024-01-16T04:23:45.136 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-16T04:23:45.137 INFO:teuthology.task.internal:Checking packages... 2024-01-16T04:23:45.160 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a70aebef70df6ec2d957347ccbfc484e0573dd5f' 2024-01-16T04:23:45.160 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-16T04:23:45.160 INFO:teuthology.packaging:ref: None 2024-01-16T04:23:45.160 INFO:teuthology.packaging:tag: None 2024-01-16T04:23:45.160 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-01-11-1300-pacific 2024-01-16T04:23:45.160 INFO:teuthology.packaging:sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T04:23:45.161 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri5-testing-2024-01-11-1300-pacific 2024-01-16T04:23:45.376 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-449.ga70aebef 2024-01-16T04:23:45.377 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-16T04:23:45.383 INFO:teuthology.task.internal:no buildpackages task found 2024-01-16T04:23:45.383 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-16T04:23:45.391 INFO:teuthology.task.internal:Saving configuration 2024-01-16T04:23:45.403 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-16T04:23:45.409 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T04:23:45.435 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 04:18:49.710715', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMIJErr9dGrhWCSEySf7+DUoB2WqQZ+JCMm6HItnXIkSTf0om2vY9E3f8+9olOrKYtE9tuLk7Llog4RR4LbdBc='} 2024-01-16T04:23:45.435 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-16T04:23:45.441 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-01-16T04:23:45.441 INFO:teuthology.run_tasks:Running task console_log... 2024-01-16T04:23:45.481 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff57122eee0>, signals=[15]) 2024-01-16T04:23:45.481 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-16T04:23:45.487 INFO:teuthology.task.internal:Opening connections... 2024-01-16T04:23:45.487 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-01-16T04:23:45.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:23:45.565 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-16T04:23:45.572 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-01-16T04:23:45.594 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-01-16T04:23:45.594 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-01-16T04:23:45.656 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-01-16T04:23:45.656 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="8" 2024-01-16T04:23:45.656 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-01-16T04:23:45.656 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-01-16T04:23:45.656 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="8" 2024-01-16T04:23:45.656 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el8" 2024-01-16T04:23:45.656 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-16T04:23:45.656 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-01-16T04:23:45.656 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-16T04:23:45.656 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-01-16T04:23:45.657 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T04:23:45.657 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-16T04:23:45.657 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T04:23:45.657 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-01-16T04:23:45.690 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-16T04:23:45.698 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-16T04:23:45.775 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-16T04:23:45.775 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-01-16T04:23:45.796 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-16T04:23:45.802 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-16T04:23:45.802 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-01-16T04:23:45.861 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-16T04:23:45.891 INFO:teuthology.run_tasks:Running task kernel... 2024-01-16T04:23:45.903 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-16T04:23:45.903 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-16T04:23:45.903 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T04:23:45.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:23:45.923 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-01-16T04:23:45.985 INFO:teuthology.orchestra.run.smithi176.stdout:4.18.0-489.el8.x86_64 2024-01-16T04:23:45.985 INFO:teuthology.task.kernel:Running kernel on smithi176: 4.18.0-489.el8.x86_64 2024-01-16T04:23:45.985 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2024-01-16T04:24:19.912 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - AppStream 1.1 MB/s | 35 MB 00:31 2024-01-16T04:24:27.753 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - BaseOS 39 MB/s | 58 MB 00:01 2024-01-16T04:24:36.202 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - Extras 90 kB/s | 18 kB 00:00 2024-01-16T04:24:36.703 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.9 kB 00:00 2024-01-16T04:24:36.979 INFO:teuthology.orchestra.run.smithi176.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2024-01-16T04:24:39.330 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 16 MB 00:02 2024-01-16T04:24:43.196 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 646 kB/s | 24 kB 00:00 2024-01-16T04:24:46.896 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-16T04:24:46.897 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-16T04:24:47.205 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:24:47.206 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:24:47.206 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-01-16T04:24:47.206 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:24:47.206 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:24:47.206 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 4.18.0-532.el8 baseos 10 M 2024-01-16T04:24:47.206 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-01-16T04:24:47.206 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core x86_64 4.18.0-532.el8 baseos 43 M 2024-01-16T04:24:47.206 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules x86_64 4.18.0-532.el8 baseos 36 M 2024-01-16T04:24:47.206 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:24:47.206 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:24:47.207 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:24:47.207 INFO:teuthology.orchestra.run.smithi176.stdout:Install 3 Packages 2024-01-16T04:24:47.207 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:24:47.207 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 90 M 2024-01-16T04:24:47.207 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 96 M 2024-01-16T04:24:47.207 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:24:47.806 INFO:teuthology.orchestra.run.smithi176.stdout:(1/3): kernel-4.18.0-532.el8.x86_64.rpm 18 MB/s | 10 MB 00:00 2024-01-16T04:24:48.856 INFO:teuthology.orchestra.run.smithi176.stdout:(2/3): kernel-core-4.18.0-532.el8.x86_64.rpm 27 MB/s | 43 MB 00:01 2024-01-16T04:24:49.215 INFO:teuthology.orchestra.run.smithi176.stdout:(3/3): kernel-modules-4.18.0-532.el8.x86_64.rpm 18 MB/s | 36 MB 00:01 2024-01-16T04:24:49.215 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:24:49.215 INFO:teuthology.orchestra.run.smithi176.stdout:Total 45 MB/s | 90 MB 00:02 2024-01-16T04:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:24:50.325 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:24:50.325 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:24:51.583 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:24:51.605 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:24:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:24:54.909 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-16T04:24:58.439 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-16T04:24:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-16T04:25:02.429 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-16T04:25:02.720 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:25:24.702 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:25:24.726 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:25:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:25:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-4.18.0-532.el8.x86_64 1/3 2024-01-16T04:25:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-core-4.18.0-532.el8.x86_64 2/3 2024-01-16T04:25:26.240 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:25:26.240 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:25:26.240 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:25:26.240 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-4.18.0-532.el8.x86_64 kernel-core-4.18.0-532.el8.x86_64 2024-01-16T04:25:26.240 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-4.18.0-532.el8.x86_64 2024-01-16T04:25:26.240 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:25:26.240 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:25:26.408 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-01-16T04:25:26.622 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-16T04:25:26.622 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-16T04:25:26.623 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-532.el8.x86_64 2024-01-16T04:25:26.623 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-16T04:25:26.623 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-16T04:25:26.623 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-16T04:25:26.623 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-01-16T04:25:26.623 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-01-16T04:25:26.834 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-16T04:25:26.834 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-16T04:25:26.834 DEBUG:teuthology.orchestra.run.smithi176:> sudo rpm -qi grub2-tools 2024-01-16T04:25:26.867 INFO:teuthology.orchestra.run.smithi176.stdout:Name : grub2-tools 2024-01-16T04:25:26.867 INFO:teuthology.orchestra.run.smithi176.stdout:Epoch : 1 2024-01-16T04:25:26.867 INFO:teuthology.orchestra.run.smithi176.stdout:Version : 2.02 2024-01-16T04:25:26.867 INFO:teuthology.orchestra.run.smithi176.stdout:Release : 129.el8 2024-01-16T04:25:26.867 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: x86_64 2024-01-16T04:25:26.867 INFO:teuthology.orchestra.run.smithi176.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-16T04:25:26.867 INFO:teuthology.orchestra.run.smithi176.stdout:Group : System Environment/Base 2024-01-16T04:25:26.867 INFO:teuthology.orchestra.run.smithi176.stdout:Size : 9265586 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:License : GPLv3+ 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:Relocations : (not relocatable) 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:Packager : CentOS BuildSystem 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:Vendor : CentOS 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:Summary : Support tools for GRUB. 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:Description : 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T04:25:26.868 INFO:teuthology.orchestra.run.smithi176.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T04:25:26.869 INFO:teuthology.orchestra.run.smithi176.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T04:25:26.869 INFO:teuthology.orchestra.run.smithi176.stdout:hardware devices. 2024-01-16T04:25:26.869 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:25:26.869 INFO:teuthology.orchestra.run.smithi176.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T04:25:26.869 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T04:25:26.870 INFO:teuthology.task.kernel:Updating grub on smithi176 to boot 4.18.0-532.el8.x86_64 2024-01-16T04:25:26.870 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T04:25:28.218 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-01-16T04:25:30.192 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-01-16T04:25:30.194 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-01-16T04:25:30.218 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.fyeN6WzQmR 2024-01-16T04:25:30.218 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fyeN6WzQmR 2024-01-16T04:25:30.292 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.fyeN6WzQmR 2024-01-16T04:25:30.432 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.fyeN6WzQmR is 6KB 2024-01-16T04:25:30.454 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.fyeN6WzQmR 2024-01-16T04:25:30.472 DEBUG:teuthology.orchestra.run.smithi176:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T04:25:30.536 INFO:teuthology.orchestra.run.smithi176.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-16T04:25:30.537 INFO:teuthology.orchestra.run.smithi176.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-16T04:25:30.537 INFO:teuthology.orchestra.run.smithi176.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-16T04:25:30.537 INFO:teuthology.orchestra.run.smithi176.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64.conf 2024-01-16T04:25:30.538 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64 2024-01-16T04:25:30.664 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-01-16T04:26:00.697 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-16T04:26:00.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-01-16T04:26:00.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:27:00.761 DEBUG:teuthology.orchestra.remote:timed out 2024-01-16T04:27:09.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-01-16T04:27:09.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:27:10.116 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-01-16T04:27:10.399 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-01-16T04:27:10.399 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-532.el8.x86_64"... 2024-01-16T04:27:10.399 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-01-16T04:27:10.415 INFO:teuthology.orchestra.run.smithi176.stdout:4.18.0-532.el8.x86_64 2024-01-16T04:27:10.416 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-532.el8.x86_64 vs 4.18.0-532.el8.x86_64 2024-01-16T04:27:10.416 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T04:27:10.416 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T04:27:11.417 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-16T04:27:11.417 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T04:27:11.488 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-01-16T04:27:11.507 DEBUG:teuthology.parallel:result is None 2024-01-16T04:27:11.508 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-16T04:27:11.515 INFO:teuthology.task.internal:Creating test directory... 2024-01-16T04:27:11.515 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T04:27:11.532 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-16T04:27:11.539 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-16T04:27:11.549 INFO:teuthology.task.internal:Creating archive directory... 2024-01-16T04:27:11.549 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T04:27:11.617 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-16T04:27:11.623 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-16T04:27:11.623 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T04:27:11.686 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T04:27:11.697 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T04:27:11.698 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-16T04:27:11.705 INFO:teuthology.task.internal:Configuring sudo... 2024-01-16T04:27:11.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T04:27:11.769 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-16T04:27:11.778 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-16T04:27:11.778 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T04:27:11.829 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T04:27:11.919 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T04:27:11.980 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T04:27:12.065 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T04:27:12.112 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:27:12.112 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T04:27:12.183 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-01-16T04:27:12.323 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T04:27:12.609 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-16T04:27:12.617 INFO:teuthology.task.internal:Starting timer... 2024-01-16T04:27:12.617 INFO:teuthology.run_tasks:Running task pcp... 2024-01-16T04:27:12.633 INFO:teuthology.run_tasks:Running task selinux... 2024-01-16T04:27:12.642 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2024-01-16T04:27:12.836 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2024-01-16T04:27:12.836 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-16T04:27:12.836 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2024-01-16T04:27:12.875 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2024-01-16T04:27:12.875 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi176.front.sepia.ceph.com': 'permissive'} 2024-01-16T04:27:12.875 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T04:27:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:27:13.278 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-16T04:27:13.278 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2024-01-16T04:27:13.880 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-16T04:27:13.891 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-16T04:27:13.907 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-16T04:27:13.908 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-16T04:32:59.485 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2024-01-16T04:32:59.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-01-16T04:32:59.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:32:59.563 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-01-16T04:32:59.631 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-01-16T04:32:59.631 INFO:teuthology.run_tasks:Running task clock... 2024-01-16T04:32:59.643 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-16T04:32:59.643 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T04:32:59.644 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T04:32:59.702 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T04:32:59.715 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T04:32:59.781 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2024-01-16T04:32:59.791 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2024-01-16T04:32:59.804 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T04:32:59.817 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T04:32:59.879 INFO:teuthology.orchestra.run.smithi176.stderr:bash: ntpq: command not found 2024-01-16T04:32:59.883 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T04:32:59.883 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-01-16T04:32:59.883 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:32:59.883 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:32:59.883 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:32:59.883 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:32:59.884 INFO:teuthology.run_tasks:Running task install... 2024-01-16T04:32:59.893 DEBUG:teuthology.task.install:project ceph 2024-01-16T04:32:59.894 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'}} 2024-01-16T04:32:59.894 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'} 2024-01-16T04:32:59.894 INFO:teuthology.task.install:Using flavor: default 2024-01-16T04:32:59.900 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-16T04:32:59.900 INFO:teuthology.task.install:extra packages: [] 2024-01-16T04:32:59.900 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'tag': None, 'wait_for_package': False} 2024-01-16T04:32:59.900 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T04:33:00.123 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-01-11-1300-pacific/a70aebef70df6ec2d957347ccbfc484e0573dd5f/centos/8/flavors/default/ 2024-01-16T04:33:00.123 INFO:teuthology.task.install.rpm:Package version is 16.2.14-449.ga70aebef 2024-01-16T04:33:00.230 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-01-11-1300-pacific/a70aebef70df6ec2d957347ccbfc484e0573dd5f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-01-11-1300-pacific/a70aebef70df6ec2d957347ccbfc484e0573dd5f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-01-11-1300-pacific/a70aebef70df6ec2d957347ccbfc484e0573dd5f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T04:33:00.230 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:33:00.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T04:33:00.258 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-16T04:33:00.258 DEBUG:teuthology.orchestra.run.smithi176:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a70aebef70df6ec2d957347ccbfc484e0573dd5f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T04:33:00.326 DEBUG:teuthology.orchestra.run.smithi176:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T04:33:00.403 DEBUG:teuthology.orchestra.run.smithi176:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T04:33:00.470 INFO:teuthology.orchestra.run.smithi176.stdout:check_obsoletes = 1 2024-01-16T04:33:00.471 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum clean all 2024-01-16T04:33:00.739 INFO:teuthology.orchestra.run.smithi176.stdout:76 files removed 2024-01-16T04:33:00.767 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw 2024-01-16T04:33:01.532 INFO:teuthology.orchestra.run.smithi176.stdout:ceph packages for x86_64 212 kB/s | 76 kB 00:00 2024-01-16T04:33:01.908 INFO:teuthology.orchestra.run.smithi176.stdout:ceph noarch packages 45 kB/s | 15 kB 00:00 2024-01-16T04:33:02.208 INFO:teuthology.orchestra.run.smithi176.stdout:ceph source packages 5.3 kB/s | 1.4 kB 00:00 2024-01-16T04:33:04.200 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - AppStream 18 MB/s | 35 MB 00:01 2024-01-16T04:33:11.959 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - BaseOS 40 MB/s | 58 MB 00:01 2024-01-16T04:33:20.559 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - Extras 52 kB/s | 18 kB 00:00 2024-01-16T04:33:21.135 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-01-16T04:33:23.328 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - PowerTools 3.3 MB/s | 6.7 MB 00:02 2024-01-16T04:33:24.787 INFO:teuthology.orchestra.run.smithi176.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-01-16T04:33:25.738 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 20 MB/s | 16 MB 00:00 2024-01-16T04:33:29.696 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 238 kB/s | 24 kB 00:00 2024-01-16T04:33:34.356 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:33:34.359 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:33:34.359 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-01-16T04:33:34.359 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:33:34.359 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:33:34.359 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw x86_64 2:16.2.14-449.ga70aebef.el8 ceph 12 M 2024-01-16T04:33:34.359 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2024-01-16T04:33:34.359 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 3.7 M 2024-01-16T04:33:34.359 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 4.2 M 2024-01-16T04:33:34.360 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-01-16T04:33:34.360 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base x86_64 2:16.2.14-449.ga70aebef.el8 ceph 6.1 M 2024-01-16T04:33:34.360 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common x86_64 2:16.2.14-449.ga70aebef.el8 ceph 24 M 2024-01-16T04:33:34.360 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux x86_64 2:16.2.14-449.ga70aebef.el8 ceph 26 k 2024-01-16T04:33:34.360 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-16T04:33:34.360 INFO:teuthology.orchestra.run.smithi176.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-16T04:33:34.360 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 825 k 2024-01-16T04:33:34.360 INFO:teuthology.orchestra.run.smithi176.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-16T04:33:34.360 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2024-01-16T04:33:34.360 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 494 k 2024-01-16T04:33:34.360 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-16T04:33:34.361 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 3.8 M 2024-01-16T04:33:34.361 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-16T04:33:34.361 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-16T04:33:34.361 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse x86_64 2:16.2.14-449.ga70aebef.el8 ceph 45 k 2024-01-16T04:33:34.361 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common x86_64 2:16.2.14-449.ga70aebef.el8 ceph 99 k 2024-01-16T04:33:34.361 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs x86_64 2:16.2.14-449.ga70aebef.el8 ceph 215 k 2024-01-16T04:33:34.361 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-16T04:33:34.361 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados x86_64 2:16.2.14-449.ga70aebef.el8 ceph 389 k 2024-01-16T04:33:34.361 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd x86_64 2:16.2.14-449.ga70aebef.el8 ceph 369 k 2024-01-16T04:33:34.361 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw x86_64 2:16.2.14-449.ga70aebef.el8 ceph 114 k 2024-01-16T04:33:34.361 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:33:34.361 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:33:34.362 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:33:34.362 INFO:teuthology.orchestra.run.smithi176.stdout:Install 21 Packages 2024-01-16T04:33:34.362 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 2 Packages 2024-01-16T04:33:34.362 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:33:34.362 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 58 M 2024-01-16T04:33:34.362 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:33:35.514 INFO:teuthology.orchestra.run.smithi176.stdout:(1/23): ceph-base-16.2.14-449.ga70aebef.el8.x86 7.9 MB/s | 6.1 MB 00:00 2024-01-16T04:33:35.572 INFO:teuthology.orchestra.run.smithi176.stdout:(2/23): ceph-selinux-16.2.14-449.ga70aebef.el8. 440 kB/s | 26 kB 00:00 2024-01-16T04:33:35.673 INFO:teuthology.orchestra.run.smithi176.stdout:(3/23): libcephfs2-16.2.14-449.ga70aebef.el8.x8 8.1 MB/s | 825 kB 00:00 2024-01-16T04:33:35.773 INFO:teuthology.orchestra.run.smithi176.stdout:(4/23): libradosstriper1-16.2.14-449.ga70aebef. 4.8 MB/s | 494 kB 00:00 2024-01-16T04:33:36.115 INFO:teuthology.orchestra.run.smithi176.stdout:(5/23): librgw2-16.2.14-449.ga70aebef.el8.x86_6 11 MB/s | 3.8 MB 00:00 2024-01-16T04:33:36.149 INFO:teuthology.orchestra.run.smithi176.stdout:(6/23): python3-ceph-argparse-16.2.14-449.ga70a 1.3 MB/s | 45 kB 00:00 2024-01-16T04:33:36.208 INFO:teuthology.orchestra.run.smithi176.stdout:(7/23): python3-ceph-common-16.2.14-449.ga70aeb 1.6 MB/s | 99 kB 00:00 2024-01-16T04:33:36.274 INFO:teuthology.orchestra.run.smithi176.stdout:(8/23): python3-cephfs-16.2.14-449.ga70aebef.el 3.2 MB/s | 215 kB 00:00 2024-01-16T04:33:36.350 INFO:teuthology.orchestra.run.smithi176.stdout:(9/23): python3-rados-16.2.14-449.ga70aebef.el8 5.1 MB/s | 389 kB 00:00 2024-01-16T04:33:36.417 INFO:teuthology.orchestra.run.smithi176.stdout:(10/23): python3-rbd-16.2.14-449.ga70aebef.el8. 5.4 MB/s | 369 kB 00:00 2024-01-16T04:33:36.475 INFO:teuthology.orchestra.run.smithi176.stdout:(11/23): python3-rgw-16.2.14-449.ga70aebef.el8. 1.9 MB/s | 114 kB 00:00 2024-01-16T04:33:36.726 INFO:teuthology.orchestra.run.smithi176.stdout:(12/23): ceph-common-16.2.14-449.ga70aebef.el8. 12 MB/s | 24 MB 00:01 2024-01-16T04:33:37.259 INFO:teuthology.orchestra.run.smithi176.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 487 kB/s | 259 kB 00:00 2024-01-16T04:33:37.293 INFO:teuthology.orchestra.run.smithi176.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 828 kB/s | 677 kB 00:00 2024-01-16T04:33:37.360 INFO:teuthology.orchestra.run.smithi176.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 706 kB/s | 47 kB 00:00 2024-01-16T04:33:37.585 INFO:teuthology.orchestra.run.smithi176.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 135 kB/s | 44 kB 00:00 2024-01-16T04:33:37.610 INFO:teuthology.orchestra.run.smithi176.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-01-16T04:33:37.669 INFO:teuthology.orchestra.run.smithi176.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2024-01-16T04:33:37.752 INFO:teuthology.orchestra.run.smithi176.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 901 kB/s | 75 kB 00:00 2024-01-16T04:33:37.803 INFO:teuthology.orchestra.run.smithi176.stdout:(20/23): leveldb-1.22-1.el8.x86_64.rpm 832 kB/s | 181 kB 00:00 2024-01-16T04:33:38.053 INFO:teuthology.orchestra.run.smithi176.stdout:(21/23): ceph-radosgw-16.2.14-449.ga70aebef.el8 3.5 MB/s | 12 MB 00:03 2024-01-16T04:33:38.120 INFO:teuthology.orchestra.run.smithi176.stdout:(22/23): librados2-16.2.14-449.ga70aebef.el8.x8 10 MB/s | 3.7 MB 00:00 2024-01-16T04:33:38.253 INFO:teuthology.orchestra.run.smithi176.stdout:(23/23): librbd1-16.2.14-449.ga70aebef.el8.x86_ 9.3 MB/s | 4.2 MB 00:00 2024-01-16T04:33:38.256 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:33:38.256 INFO:teuthology.orchestra.run.smithi176.stdout:Total 15 MB/s | 58 MB 00:03 2024-01-16T04:33:38.299 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:33:38.347 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:33:38.347 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:33:38.680 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:33:38.683 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:33:39.722 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:33:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-01-16T04:33:40.114 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-01-16T04:33:40.504 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-01-16T04:33:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librados2-2:16.2.14-449.ga70aebef.el8.x86_64 3/25 2024-01-16T04:33:40.744 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:16.2.14-449.ga70aebef.el8.x86_64 3/25 2024-01-16T04:33:41.164 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-2:16.2.14-449.ga70aebef.el8.x86_64 4/25 2024-01-16T04:33:41.339 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 5/25 2024-01-16T04:33:41.755 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 5/25 2024-01-16T04:33:41.922 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 6/25 2024-01-16T04:33:41.992 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 6/25 2024-01-16T04:33:42.207 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2024-01-16T04:33:42.735 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-01-16T04:33:42.889 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librgw2-2:16.2.14-449.ga70aebef.el8.x86_64 9/25 2024-01-16T04:33:42.948 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librgw2-2:16.2.14-449.ga70aebef.el8.x86_64 9/25 2024-01-16T04:33:43.136 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-01-16T04:33:43.311 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-argparse-2:16.2.14-449.ga70aebef.el8. 11/25 2024-01-16T04:33:43.492 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-2:16.2.14-449.ga70aebef.el8.x86_64 12/25 2024-01-16T04:33:43.747 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-2:16.2.14-449.ga70aebef.el8.x86_64 13/25 2024-01-16T04:33:43.964 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-2:16.2.14-449.ga70aebef.el8.x86_64 14/25 2024-01-16T04:33:44.105 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libradosstriper1-2:16.2.14-449.ga70aebef.el8.x86_6 15/25 2024-01-16T04:33:44.160 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libradosstriper1-2:16.2.14-449.ga70aebef.el8.x86_6 15/25 2024-01-16T04:33:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-01-16T04:33:44.546 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-01-16T04:33:44.732 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-01-16T04:33:44.922 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-common-2:16.2.14-449.ga70aebef.el8.x8 19/25 2024-01-16T04:33:47.429 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 20/25 2024-01-16T04:33:47.655 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 20/25 2024-01-16T04:33:48.342 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 20/25 2024-01-16T04:33:48.639 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 21/25 2024-01-16T04:33:48.748 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 21/25 2024-01-16T04:33:48.871 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-selinux-2:16.2.14-449.ga70aebef.el8.x86_64 22/25 2024-01-16T04:34:07.695 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-selinux-2:16.2.14-449.ga70aebef.el8.x86_64 22/25 2024-01-16T04:34:07.696 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /sys 2024-01-16T04:34:07.696 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /proc 2024-01-16T04:34:07.696 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev 2024-01-16T04:34:07.696 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run 2024-01-16T04:34:07.696 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2024-01-16T04:34:07.696 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /var/tmp 2024-01-16T04:34:07.696 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /home 2024-01-16T04:34:07.696 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /tmp 2024-01-16T04:34:07.696 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev 2024-01-16T04:34:07.696 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:08.852 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-2:16.2.14-449.ga70aebef.el8.x86_64 23/25 2024-01-16T04:34:08.915 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-449.ga70aebef.el8.x86_64 23/25 2024-01-16T04:34:08.915 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-01-16T04:34:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-01-16T04:34:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-01-16T04:34:09.783 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-01-16T04:34:09.783 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 1/25 2024-01-16T04:34:09.783 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 2/25 2024-01-16T04:34:09.783 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-2:16.2.14-449.ga70aebef.el8.x86_64 3/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-selinux-2:16.2.14-449.ga70aebef.el8.x86_64 4/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 5/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libradosstriper1-2:16.2.14-449.ga70aebef.el8.x86_6 6/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librgw2-2:16.2.14-449.ga70aebef.el8.x86_64 7/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-argparse-2:16.2.14-449.ga70aebef.el8. 8/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-common-2:16.2.14-449.ga70aebef.el8.x8 9/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-2:16.2.14-449.ga70aebef.el8.x86_64 10/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-2:16.2.14-449.ga70aebef.el8.x86_64 11/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-2:16.2.14-449.ga70aebef.el8.x86_64 12/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-2:16.2.14-449.ga70aebef.el8.x86_64 13/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-01-16T04:34:09.784 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-01-16T04:34:09.785 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2024-01-16T04:34:09.785 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-01-16T04:34:09.785 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-01-16T04:34:09.785 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-01-16T04:34:09.785 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-01-16T04:34:09.785 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:16.2.14-449.ga70aebef.el8.x86_64 22/25 2024-01-16T04:34:09.785 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-01-16T04:34:09.785 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 24/25 2024-01-16T04:34:10.905 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-01-16T04:34:10.905 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:10.905 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2024-01-16T04:34:10.905 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.905 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: leveldb-1.22-1.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-16T04:34:10.906 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-16T04:34:10.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-16T04:34:10.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.907 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:10.907 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:10.907 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:34:11.180 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-test 2024-01-16T04:34:12.038 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:43 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:34:13.747 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:34:13.748 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:34:13.748 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-01-16T04:34:13.748 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:34:13.748 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:34:13.748 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test x86_64 2:16.2.14-449.ga70aebef.el8 ceph 52 M 2024-01-16T04:34:13.748 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-01-16T04:34:13.748 INFO:teuthology.orchestra.run.smithi176.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-16T04:34:13.748 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite x86_64 2:16.2.14-449.ga70aebef.el8 ceph 174 k 2024-01-16T04:34:13.749 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2024-01-16T04:34:13.749 INFO:teuthology.orchestra.run.smithi176.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-16T04:34:13.749 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-16T04:34:13.749 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:13.749 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:34:13.749 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:34:13.749 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-01-16T04:34:13.749 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:13.750 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 53 M 2024-01-16T04:34:13.750 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 196 M 2024-01-16T04:34:13.750 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:34:13.989 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): libcephsqlite-16.2.14-449.ga70aebef.el8. 885 kB/s | 174 kB 00:00 2024-01-16T04:34:14.123 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 615 kB/s | 203 kB 00:00 2024-01-16T04:34:14.281 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 642 kB/s | 187 kB 00:00 2024-01-16T04:34:14.365 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.3 MB/s | 323 kB 00:00 2024-01-16T04:34:14.390 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 650 kB/s | 71 kB 00:00 2024-01-16T04:34:16.315 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): ceph-test-16.2.14-449.ga70aebef.el8.x86_ 21 MB/s | 52 MB 00:02 2024-01-16T04:34:16.317 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:34:16.318 INFO:teuthology.orchestra.run.smithi176.stdout:Total 21 MB/s | 53 MB 00:02 2024-01-16T04:34:16.361 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:34:16.381 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:34:16.381 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:34:16.696 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:34:16.699 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:34:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:34:17.396 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-16T04:34:17.628 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-16T04:34:17.792 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-16T04:34:17.855 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-16T04:34:18.048 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-16T04:34:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephsqlite-2:16.2.14-449.ga70aebef.el8.x86_64 5/6 2024-01-16T04:34:22.768 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephsqlite-2:16.2.14-449.ga70aebef.el8.x86_64 5/6 2024-01-16T04:34:22.973 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-2:16.2.14-449.ga70aebef.el8.x86_64 6/6 2024-01-16T04:34:23.256 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-test-2:16.2.14-449.ga70aebef.el8.x86_64 6/6 2024-01-16T04:34:23.257 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-2:16.2.14-449.ga70aebef.el8.x86_64 1/6 2024-01-16T04:34:23.257 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephsqlite-2:16.2.14-449.ga70aebef.el8.x86_64 2/6 2024-01-16T04:34:23.257 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-16T04:34:23.257 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2024-01-16T04:34:23.257 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-16T04:34:24.009 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-16T04:34:24.009 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:24.009 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:34:24.009 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:24.009 INFO:teuthology.orchestra.run.smithi176.stdout: jq-1.6-8.el8.x86_64 2024-01-16T04:34:24.009 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:24.009 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-16T04:34:24.009 INFO:teuthology.orchestra.run.smithi176.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-16T04:34:24.009 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-16T04:34:24.009 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:24.009 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:34:24.208 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph 2024-01-16T04:34:25.066 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:56 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:34:26.780 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:34:26.783 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:34:26.784 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-01-16T04:34:26.784 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:34:26.784 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:34:26.784 INFO:teuthology.orchestra.run.smithi176.stdout: ceph x86_64 2:16.2.14-449.ga70aebef.el8 ceph 6.4 k 2024-01-16T04:34:26.784 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-01-16T04:34:26.784 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds x86_64 2:16.2.14-449.ga70aebef.el8 ceph 2.4 M 2024-01-16T04:34:26.784 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr x86_64 2:16.2.14-449.ga70aebef.el8 ceph 1.6 M 2024-01-16T04:34:26.784 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 245 k 2024-01-16T04:34:26.784 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon x86_64 2:16.2.14-449.ga70aebef.el8 ceph 5.0 M 2024-01-16T04:34:26.784 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd x86_64 2:16.2.14-449.ga70aebef.el8 ceph 18 M 2024-01-16T04:34:26.784 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-16T04:34:26.784 INFO:teuthology.orchestra.run.smithi176.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-16T04:34:26.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:34:26.786 INFO:teuthology.orchestra.run.smithi176.stdout:Install 32 Packages 2024-01-16T04:34:26.787 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:26.787 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 31 M 2024-01-16T04:34:26.787 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 101 M 2024-01-16T04:34:26.787 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:34:27.241 INFO:teuthology.orchestra.run.smithi176.stdout:(1/32): ceph-16.2.14-449.ga70aebef.el8.x86_64.r 33 kB/s | 6.4 kB 00:00 2024-01-16T04:34:27.366 INFO:teuthology.orchestra.run.smithi176.stdout:(2/32): ceph-mgr-16.2.14-449.ga70aebef.el8.x86_ 5.0 MB/s | 1.6 MB 00:00 2024-01-16T04:34:27.767 INFO:teuthology.orchestra.run.smithi176.stdout:(3/32): ceph-mds-16.2.14-449.ga70aebef.el8.x86_ 3.4 MB/s | 2.4 MB 00:00 2024-01-16T04:34:27.875 INFO:teuthology.orchestra.run.smithi176.stdout:(4/32): ceph-mgr-modules-core-16.2.14-449.ga70a 2.2 MB/s | 245 kB 00:00 2024-01-16T04:34:28.150 INFO:teuthology.orchestra.run.smithi176.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 571 kB/s | 157 kB 00:00 2024-01-16T04:34:28.267 INFO:teuthology.orchestra.run.smithi176.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 332 kB/s | 39 kB 00:00 2024-01-16T04:34:28.476 INFO:teuthology.orchestra.run.smithi176.stdout:(7/32): ceph-osd-16.2.14-449.ga70aebef.el8.x86_ 17 MB/s | 18 MB 00:01 2024-01-16T04:34:28.502 INFO:teuthology.orchestra.run.smithi176.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 442 kB/s | 103 kB 00:00 2024-01-16T04:34:28.635 INFO:teuthology.orchestra.run.smithi176.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 360 kB/s | 57 kB 00:00 2024-01-16T04:34:28.702 INFO:teuthology.orchestra.run.smithi176.stdout:(10/32): ceph-mon-16.2.14-449.ga70aebef.el8.x86 3.4 MB/s | 5.0 MB 00:01 2024-01-16T04:34:28.727 INFO:teuthology.orchestra.run.smithi176.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.0 MB/s | 457 kB 00:00 2024-01-16T04:34:28.786 INFO:teuthology.orchestra.run.smithi176.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.6 MB/s | 249 kB 00:00 2024-01-16T04:34:28.811 INFO:teuthology.orchestra.run.smithi176.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.3 MB/s | 109 kB 00:00 2024-01-16T04:34:28.878 INFO:teuthology.orchestra.run.smithi176.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 3.6 MB/s | 641 kB 00:00 2024-01-16T04:34:29.220 INFO:teuthology.orchestra.run.smithi176.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 100 kB/s | 44 kB 00:00 2024-01-16T04:34:29.379 INFO:teuthology.orchestra.run.smithi176.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 326 kB/s | 185 kB 00:00 2024-01-16T04:34:29.429 INFO:teuthology.orchestra.run.smithi176.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 314 kB/s | 173 kB 00:00 2024-01-16T04:34:29.504 INFO:teuthology.orchestra.run.smithi176.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2024-01-16T04:34:29.546 INFO:teuthology.orchestra.run.smithi176.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 150 kB/s | 18 kB 00:00 2024-01-16T04:34:29.580 INFO:teuthology.orchestra.run.smithi176.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.0 MB/s | 384 kB 00:00 2024-01-16T04:34:29.630 INFO:teuthology.orchestra.run.smithi176.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 388 kB/s | 49 kB 00:00 2024-01-16T04:34:29.680 INFO:teuthology.orchestra.run.smithi176.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 439 kB/s | 59 kB 00:00 2024-01-16T04:34:29.730 INFO:teuthology.orchestra.run.smithi176.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2024-01-16T04:34:29.756 INFO:teuthology.orchestra.run.smithi176.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 124 kB/s | 16 kB 00:00 2024-01-16T04:34:29.798 INFO:teuthology.orchestra.run.smithi176.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 160 kB/s | 19 kB 00:00 2024-01-16T04:34:29.840 INFO:teuthology.orchestra.run.smithi176.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2024-01-16T04:34:29.873 INFO:teuthology.orchestra.run.smithi176.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 249 kB/s | 29 kB 00:00 2024-01-16T04:34:29.915 INFO:teuthology.orchestra.run.smithi176.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 235 kB/s | 27 kB 00:00 2024-01-16T04:34:29.982 INFO:teuthology.orchestra.run.smithi176.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.8 MB/s | 255 kB 00:00 2024-01-16T04:34:30.057 INFO:teuthology.orchestra.run.smithi176.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 601 kB/s | 85 kB 00:00 2024-01-16T04:34:30.082 INFO:teuthology.orchestra.run.smithi176.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.2 MB/s | 251 kB 00:00 2024-01-16T04:34:30.107 INFO:teuthology.orchestra.run.smithi176.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2024-01-16T04:34:30.112 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:34:30.112 INFO:teuthology.orchestra.run.smithi176.stdout:Total 9.4 MB/s | 31 MB 00:03 2024-01-16T04:34:30.197 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:34:30.235 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:34:30.236 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:34:30.538 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:34:30.541 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:34:31.647 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:34:31.845 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-16T04:34:32.039 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-16T04:34:32.222 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-16T04:34:32.462 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-16T04:34:32.660 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-16T04:34:32.849 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-16T04:34:33.034 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-16T04:34:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-16T04:34:33.482 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-16T04:34:33.733 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-16T04:34:33.979 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-16T04:34:34.211 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-16T04:34:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-16T04:34:34.850 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-16T04:34:35.030 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-16T04:34:35.246 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-16T04:34:35.473 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-16T04:34:35.775 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-16T04:34:35.987 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-16T04:34:36.199 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-16T04:34:36.394 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-16T04:34:36.669 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-16T04:34:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-16T04:34:37.108 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-16T04:34:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-16T04:34:37.724 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-16T04:34:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-449.ga70aebef.el8. 27/32 2024-01-16T04:34:38.239 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 28/32 2024-01-16T04:34:39.736 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 28/32 2024-01-16T04:34:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-2:16.2.14-449.ga70aebef.el8.x86_64 29/32 2024-01-16T04:34:40.345 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-2:16.2.14-449.ga70aebef.el8.x86_64 29/32 2024-01-16T04:34:40.513 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-2:16.2.14-449.ga70aebef.el8.x86_64 30/32 2024-01-16T04:34:40.730 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-2:16.2.14-449.ga70aebef.el8.x86_64 30/32 2024-01-16T04:34:40.876 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-2:16.2.14-449.ga70aebef.el8.x86_64 31/32 2024-01-16T04:34:40.894 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-2:16.2.14-449.ga70aebef.el8.x86_64 31/32 2024-01-16T04:34:41.013 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-2:16.2.14-449.ga70aebef.el8.x86_64 32/32 2024-01-16T04:34:41.437 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-2:16.2.14-449.ga70aebef.el8.x86_64 32/32 2024-01-16T04:34:41.437 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-2:16.2.14-449.ga70aebef.el8.x86_64 1/32 2024-01-16T04:34:41.437 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-2:16.2.14-449.ga70aebef.el8.x86_64 2/32 2024-01-16T04:34:41.437 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 3/32 2024-01-16T04:34:41.437 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-2:16.2.14-449.ga70aebef.el8.x86_64 4/32 2024-01-16T04:34:41.437 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-2:16.2.14-449.ga70aebef.el8.x86_64 5/32 2024-01-16T04:34:41.437 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-449.ga70aebef.el8. 6/32 2024-01-16T04:34:41.437 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-16T04:34:41.437 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-16T04:34:41.437 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-16T04:34:41.437 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-16T04:34:41.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-16T04:34:41.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-16T04:34:41.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-16T04:34:41.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-16T04:34:41.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-16T04:34:41.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-16T04:34:41.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-16T04:34:41.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-16T04:34:41.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-16T04:34:42.797 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-16T04:34:42.797 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:42.797 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:34:42.797 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-16T04:34:42.798 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend-2.6-1.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-16T04:34:42.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-16T04:34:42.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-16T04:34:42.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-16T04:34:42.800 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:42.800 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:34:43.025 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-base 2024-01-16T04:34:43.896 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:14 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:34:45.374 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:34:45.611 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:34:45.612 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-01-16T04:34:45.612 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:34:45.720 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install cephadm 2024-01-16T04:34:46.573 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:17 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:34:48.315 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:34:48.316 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:34:48.316 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-01-16T04:34:48.316 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:34:48.316 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:34:48.316 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 85 k 2024-01-16T04:34:48.316 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:48.316 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:34:48.316 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:34:48.316 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-01-16T04:34:48.316 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:48.316 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 85 k 2024-01-16T04:34:48.317 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 352 k 2024-01-16T04:34:48.317 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:34:48.481 INFO:teuthology.orchestra.run.smithi176.stdout:cephadm-16.2.14-449.ga70aebef.el8.noarch.rpm 518 kB/s | 85 kB 00:00 2024-01-16T04:34:48.482 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:34:48.482 INFO:teuthology.orchestra.run.smithi176.stdout:Total 514 kB/s | 85 kB 00:00 2024-01-16T04:34:48.482 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:34:48.489 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:34:48.489 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:34:48.491 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:34:48.494 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:34:48.537 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:34:49.023 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/1 2024-01-16T04:34:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/1 2024-01-16T04:34:49.423 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/1 2024-01-16T04:34:50.127 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/1 2024-01-16T04:34:50.127 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:50.127 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:34:50.127 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:34:50.127 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:50.127 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:34:50.344 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-immutable-object-cache 2024-01-16T04:34:51.205 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:22 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:34:52.929 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:34:52.930 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:34:52.930 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-01-16T04:34:52.930 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:34:52.931 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:34:52.931 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-449.ga70aebef.el8 ceph 161 k 2024-01-16T04:34:52.931 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:52.931 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:34:52.931 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:34:52.931 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-01-16T04:34:52.931 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:52.931 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 161 k 2024-01-16T04:34:52.931 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 470 k 2024-01-16T04:34:52.931 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:34:53.930 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-immutable-object-cache-16.2.14-449.ga70aeb 161 kB/s | 161 kB 00:00 2024-01-16T04:34:53.932 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:34:53.932 INFO:teuthology.orchestra.run.smithi176.stdout:Total 161 kB/s | 161 kB 00:01 2024-01-16T04:34:53.935 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:34:53.944 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:34:53.944 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:34:53.994 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:34:53.997 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:34:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:34:54.778 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-449.ga70aebef. 1/1 2024-01-16T04:34:55.194 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-449.ga70aebef. 1/1 2024-01-16T04:34:55.889 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-449.ga70aebef. 1/1 2024-01-16T04:34:55.890 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:55.890 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:34:55.890 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:34:55.890 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:34:55.890 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:34:56.112 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr 2024-01-16T04:34:56.970 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:27 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:34:58.448 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:34:58.685 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:34:58.686 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-01-16T04:34:58.686 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:34:58.794 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-dashboard 2024-01-16T04:34:59.642 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:30 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:35:01.354 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:35:01.355 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:01.355 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-01-16T04:35:01.355 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:01.355 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:35:01.355 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 3.1 M 2024-01-16T04:35:01.355 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-01-16T04:35:01.355 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 24 k 2024-01-16T04:35:01.355 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 15 k 2024-01-16T04:35:01.355 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-16T04:35:01.355 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-16T04:35:01.355 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-16T04:35:01.355 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:01.356 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:35:01.356 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:01.356 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-01-16T04:35:01.356 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:01.356 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 3.4 M 2024-01-16T04:35:01.356 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 22 M 2024-01-16T04:35:01.357 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:35:01.746 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): ceph-grafana-dashboards-16.2.14-449.ga70 169 kB/s | 24 kB 00:00 2024-01-16T04:35:01.938 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): python3-jwt-1.6.1-2.el8.noarch.rpm 223 kB/s | 43 kB 00:00 2024-01-16T04:35:01.963 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): ceph-prometheus-alerts-16.2.14-449.ga70a 41 kB/s | 15 kB 00:00 2024-01-16T04:35:02.297 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 94 kB/s | 34 kB 00:00 2024-01-16T04:35:02.506 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 362 kB/s | 196 kB 00:00 2024-01-16T04:35:02.614 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): ceph-mgr-dashboard-16.2.14-449.ga70aebef 3.1 MB/s | 3.1 MB 00:01 2024-01-16T04:35:02.616 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:35:02.616 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.7 MB/s | 3.4 MB 00:01 2024-01-16T04:35:02.628 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:35:02.638 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:35:02.638 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:35:02.792 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:35:02.795 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:35:03.065 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:35:03.257 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-16T04:35:03.501 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-16T04:35:03.678 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-16T04:35:03.799 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-449.ga70aebef.el8.n 4/6 2024-01-16T04:35:04.808 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-449.ga70aebef.el8. 5/6 2024-01-16T04:35:05.093 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-449.ga70aebef.el8.noarc 6/6 2024-01-16T04:35:05.381 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-449.ga70aebef.el8.noarc 6/6 2024-01-16T04:35:05.381 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-449.ga70aebef.el8. 1/6 2024-01-16T04:35:05.382 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-449.ga70aebef.el8.noarc 2/6 2024-01-16T04:35:05.382 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-449.ga70aebef.el8.n 3/6 2024-01-16T04:35:05.382 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-16T04:35:05.382 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-16T04:35:06.094 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-16T04:35:06.094 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:06.094 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:35:06.094 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:35:06.094 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:35:06.094 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:35:06.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-16T04:35:06.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-16T04:35:06.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-16T04:35:06.095 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:06.095 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:35:06.228 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-16T04:35:07.083 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:38 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:35:08.796 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:35:08.798 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-01-16T04:35:08.798 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-01-16T04:35:08.798 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-01-16T04:35:08.798 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:35:08.798 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 8.6 M 2024-01-16T04:35:08.798 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-01-16T04:35:08.798 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-16T04:35:08.798 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-16T04:35:08.798 INFO:teuthology.orchestra.run.smithi176.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-16T04:35:08.799 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-16T04:35:08.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-16T04:35:08.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-16T04:35:08.799 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-16T04:35:08.799 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:08.799 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:35:08.799 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-01-16T04:35:08.799 INFO:teuthology.orchestra.run.smithi176.stdout:Install 8 Packages 2024-01-16T04:35:08.799 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:08.800 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 37 M 2024-01-16T04:35:08.800 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 210 M 2024-01-16T04:35:08.800 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:35:10.010 INFO:teuthology.orchestra.run.smithi176.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-44 11 MB/s | 8.6 MB 00:00 2024-01-16T04:35:10.436 INFO:teuthology.orchestra.run.smithi176.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 3.9 MB/s | 4.7 MB 00:01 2024-01-16T04:35:10.810 INFO:teuthology.orchestra.run.smithi176.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.1 MB/s | 4.9 MB 00:01 2024-01-16T04:35:10.979 INFO:teuthology.orchestra.run.smithi176.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 415 kB/s | 225 kB 00:00 2024-01-16T04:35:11.131 INFO:teuthology.orchestra.run.smithi176.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 4.2 MB/s | 645 kB 00:00 2024-01-16T04:35:11.171 INFO:teuthology.orchestra.run.smithi176.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 4.1 MB/s | 172 kB 00:00 2024-01-16T04:35:11.630 INFO:teuthology.orchestra.run.smithi176.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 2.3 MB/s | 3.7 MB 00:01 2024-01-16T04:35:12.280 INFO:teuthology.orchestra.run.smithi176.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 9.6 MB/s | 14 MB 00:01 2024-01-16T04:35:12.280 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:35:12.281 INFO:teuthology.orchestra.run.smithi176.stdout:Total 11 MB/s | 37 MB 00:03 2024-01-16T04:35:12.516 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:35:12.535 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:35:12.535 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:35:12.852 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:35:12.855 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:35:13.437 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:35:13.582 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-16T04:35:13.714 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-16T04:35:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-16T04:35:14.454 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-16T04:35:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-16T04:35:15.259 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-16T04:35:15.407 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-16T04:35:16.137 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-16T04:35:16.491 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-16T04:35:16.740 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-16T04:35:18.595 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-16T04:35:19.902 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-16T04:35:20.098 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-449.ga70aebe 8/8 2024-01-16T04:35:20.427 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-449.ga70aebe 8/8 2024-01-16T04:35:20.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-449.ga70aebe 1/8 2024-01-16T04:35:20.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-16T04:35:20.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-16T04:35:20.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-16T04:35:20.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-16T04:35:20.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-16T04:35:20.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-16T04:35:21.154 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-16T04:35:21.154 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:21.154 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:35:21.154 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:35:21.154 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-16T04:35:21.154 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-16T04:35:21.154 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-16T04:35:21.154 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-16T04:35:21.154 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-16T04:35:21.155 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-16T04:35:21.155 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-16T04:35:21.155 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:21.155 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:35:21.290 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-rook 2024-01-16T04:35:22.144 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:53 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:35:23.860 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:35:23.862 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:23.862 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-01-16T04:35:23.862 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:23.862 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:35:23.862 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 27 k 2024-01-16T04:35:23.862 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:23.863 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:35:23.864 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:23.864 INFO:teuthology.orchestra.run.smithi176.stdout:Install 13 Packages 2024-01-16T04:35:23.864 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:23.864 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 1.9 M 2024-01-16T04:35:23.864 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 24 M 2024-01-16T04:35:23.864 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:35:24.544 INFO:teuthology.orchestra.run.smithi176.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 116 kB/s | 27 kB 00:00 2024-01-16T04:35:24.569 INFO:teuthology.orchestra.run.smithi176.stdout:(2/13): ceph-mgr-rook-16.2.14-449.ga70aebef.el8 108 kB/s | 27 kB 00:00 2024-01-16T04:35:24.594 INFO:teuthology.orchestra.run.smithi176.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 71 kB/s | 20 kB 00:00 2024-01-16T04:35:24.753 INFO:teuthology.orchestra.run.smithi176.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 978 kB/s | 155 kB 00:00 2024-01-16T04:35:24.795 INFO:teuthology.orchestra.run.smithi176.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 504 kB/s | 126 kB 00:00 2024-01-16T04:35:24.811 INFO:teuthology.orchestra.run.smithi176.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 728 kB/s | 43 kB 00:00 2024-01-16T04:35:24.887 INFO:teuthology.orchestra.run.smithi176.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 346 kB/s | 110 kB 00:00 2024-01-16T04:35:25.121 INFO:teuthology.orchestra.run.smithi176.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 101 kB/s | 33 kB 00:00 2024-01-16T04:35:25.138 INFO:teuthology.orchestra.run.smithi176.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 50 kB/s | 16 kB 00:00 2024-01-16T04:35:25.254 INFO:teuthology.orchestra.run.smithi176.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 221 kB/s | 81 kB 00:00 2024-01-16T04:35:25.363 INFO:teuthology.orchestra.run.smithi176.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 285 kB/s | 64 kB 00:00 2024-01-16T04:35:25.380 INFO:teuthology.orchestra.run.smithi176.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 489 kB/s | 61 kB 00:00 2024-01-16T04:35:25.613 INFO:teuthology.orchestra.run.smithi176.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.4 MB/s | 1.2 MB 00:00 2024-01-16T04:35:25.614 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:35:25.614 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2024-01-16T04:35:25.672 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:35:25.688 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:35:25.688 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:35:25.775 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:35:25.778 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:35:26.266 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:35:26.541 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-16T04:35:26.739 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-16T04:35:26.966 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-16T04:35:27.187 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-16T04:35:27.395 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-16T04:35:27.610 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-16T04:35:27.890 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-16T04:35:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-16T04:35:29.185 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-16T04:35:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-16T04:35:29.637 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-16T04:35:29.796 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-16T04:35:29.946 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-rook-2:16.2.14-449.ga70aebef.el8.noarch 13/13 2024-01-16T04:35:30.222 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-449.ga70aebef.el8.noarch 13/13 2024-01-16T04:35:30.222 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-rook-2:16.2.14-449.ga70aebef.el8.noarch 1/13 2024-01-16T04:35:30.222 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-16T04:35:30.222 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-16T04:35:30.222 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-16T04:35:30.223 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-16T04:35:30.223 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-16T04:35:30.223 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-16T04:35:30.223 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-16T04:35:30.223 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-16T04:35:30.223 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-16T04:35:30.223 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-16T04:35:30.223 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-16T04:35:30.999 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-16T04:35:30.999 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-16T04:35:31.000 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-16T04:35:31.001 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-16T04:35:31.001 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:31.001 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:35:31.164 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-cephadm 2024-01-16T04:35:32.012 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:03 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:35:33.741 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:35:33.743 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:33.743 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-01-16T04:35:33.743 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:33.743 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:35:33.743 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 116 k 2024-01-16T04:35:33.743 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-01-16T04:35:33.743 INFO:teuthology.orchestra.run.smithi176.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-16T04:35:33.743 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-16T04:35:33.743 INFO:teuthology.orchestra.run.smithi176.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-16T04:35:33.743 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-16T04:35:33.743 INFO:teuthology.orchestra.run.smithi176.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-16T04:35:33.744 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:33.744 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:35:33.744 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:33.744 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-01-16T04:35:33.744 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:33.744 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 5.8 M 2024-01-16T04:35:33.745 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 24 M 2024-01-16T04:35:33.745 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:35:34.404 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): ceph-mgr-cephadm-16.2.14-449.ga70aebef.e 398 kB/s | 116 kB 00:00 2024-01-16T04:35:34.429 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.7 MB/s | 538 kB 00:00 2024-01-16T04:35:34.513 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 12 MB/s | 4.8 MB 00:00 2024-01-16T04:35:34.538 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 158 kB/s | 21 kB 00:00 2024-01-16T04:35:34.647 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 366 kB/s | 49 kB 00:00 2024-01-16T04:35:34.705 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.3 MB/s | 355 kB 00:00 2024-01-16T04:35:34.706 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:35:34.706 INFO:teuthology.orchestra.run.smithi176.stdout:Total 6.1 MB/s | 5.8 MB 00:00 2024-01-16T04:35:34.757 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:35:34.767 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:35:34.767 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:35:34.873 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:35:34.876 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:35:35.211 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:35:35.461 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-16T04:35:35.715 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-16T04:35:36.622 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-16T04:35:37.300 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-16T04:35:37.553 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-16T04:35:37.745 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-449.ga70aebef.el8.noarch 6/6 2024-01-16T04:35:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-449.ga70aebef.el8.noarch 6/6 2024-01-16T04:35:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/6 2024-01-16T04:35:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-16T04:35:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-16T04:35:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-16T04:35:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-16T04:35:38.735 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-16T04:35:38.735 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:38.735 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:35:38.735 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:35:38.735 INFO:teuthology.orchestra.run.smithi176.stdout: python3-apipkg-1.5-6.el8.noarch 2024-01-16T04:35:38.735 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-16T04:35:38.735 INFO:teuthology.orchestra.run.smithi176.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-01-16T04:35:38.735 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-16T04:35:38.735 INFO:teuthology.orchestra.run.smithi176.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-01-16T04:35:38.735 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:38.735 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:35:38.846 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-fuse 2024-01-16T04:35:39.704 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:10 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:35:41.421 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:35:41.422 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:41.422 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-01-16T04:35:41.422 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:41.422 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:35:41.422 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse x86_64 2:16.2.14-449.ga70aebef.el8 ceph 911 k 2024-01-16T04:35:41.422 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-01-16T04:35:41.422 INFO:teuthology.orchestra.run.smithi176.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-16T04:35:41.422 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:41.422 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:35:41.422 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:41.423 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2024-01-16T04:35:41.423 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:41.423 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 994 k 2024-01-16T04:35:41.423 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 2.8 M 2024-01-16T04:35:41.423 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:35:41.698 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 938 kB/s | 83 kB 00:00 2024-01-16T04:35:41.874 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-fuse-16.2.14-449.ga70aebef.el8.x86_ 3.4 MB/s | 911 kB 00:00 2024-01-16T04:35:41.874 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:35:41.874 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.2 MB/s | 994 kB 00:00 2024-01-16T04:35:41.884 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:35:41.905 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:35:41.905 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:35:41.991 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:35:41.994 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:35:42.435 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-16T04:35:42.627 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 2/2 2024-01-16T04:35:43.005 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 2/2 2024-01-16T04:35:43.005 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 1/2 2024-01-16T04:35:43.719 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-16T04:35:43.719 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:43.719 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:35:43.719 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-16T04:35:43.719 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:43.720 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:35:43.922 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados-devel 2024-01-16T04:35:44.784 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:15 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:35:46.493 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:35:46.493 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-01-16T04:35:46.494 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:46.494 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:35:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel x86_64 2:16.2.14-449.ga70aebef.el8 ceph 131 k 2024-01-16T04:35:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:46.494 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:35:46.494 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:46.494 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-01-16T04:35:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:46.495 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 131 k 2024-01-16T04:35:46.495 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 432 k 2024-01-16T04:35:46.495 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:35:46.685 INFO:teuthology.orchestra.run.smithi176.stdout:librados-devel-16.2.14-449.ga70aebef.el8.x86_64 690 kB/s | 131 kB 00:00 2024-01-16T04:35:46.686 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:35:46.686 INFO:teuthology.orchestra.run.smithi176.stdout:Total 686 kB/s | 131 kB 00:00 2024-01-16T04:35:46.686 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:35:46.691 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:35:46.691 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:35:46.741 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:35:46.744 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:35:46.851 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:35:47.067 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:35:47.352 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:35:48.707 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:35:48.707 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:48.707 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:35:48.707 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:35:48.707 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:48.707 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:35:49.889 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs2 2024-01-16T04:35:50.761 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:21 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:35:52.241 INFO:teuthology.orchestra.run.smithi176.stdout:Package libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:35:52.476 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:35:52.477 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-01-16T04:35:52.477 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:35:52.581 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs-devel 2024-01-16T04:35:53.438 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:24 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:35:55.154 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:35:55.154 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:55.154 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-01-16T04:35:55.155 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:55.155 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:35:55.155 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel x86_64 2:16.2.14-449.ga70aebef.el8 ceph 25 k 2024-01-16T04:35:55.155 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:55.155 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:35:55.155 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:35:55.155 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-01-16T04:35:55.155 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:55.156 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 25 k 2024-01-16T04:35:55.156 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 107 k 2024-01-16T04:35:55.156 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:35:55.442 INFO:teuthology.orchestra.run.smithi176.stdout:libcephfs-devel-16.2.14-449.ga70aebef.el8.x86_6 89 kB/s | 25 kB 00:00 2024-01-16T04:35:55.443 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:35:55.443 INFO:teuthology.orchestra.run.smithi176.stdout:Total 89 kB/s | 25 kB 00:00 2024-01-16T04:35:55.443 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:35:55.447 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:35:55.448 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:35:55.450 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:35:55.453 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:35:55.505 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:35:55.691 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:35:55.958 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:35:56.644 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:35:56.644 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:56.644 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:35:56.644 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:35:56.645 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:35:56.645 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:35:56.765 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados2 2024-01-16T04:35:57.617 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:28 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:35:59.099 INFO:teuthology.orchestra.run.smithi176.stdout:Package librados2-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:35:59.333 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:35:59.334 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-01-16T04:35:59.334 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:35:59.436 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librbd1 2024-01-16T04:36:00.300 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:31 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:36:01.779 INFO:teuthology.orchestra.run.smithi176.stdout:Package librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:36:02.013 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:36:02.014 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-01-16T04:36:02.014 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:36:02.109 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rados 2024-01-16T04:36:02.970 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:33 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:36:04.455 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rados-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:36:04.691 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:36:04.692 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-01-16T04:36:04.692 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:36:04.800 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rgw 2024-01-16T04:36:05.668 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:36 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:36:07.138 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rgw-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:36:07.373 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:36:07.374 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-01-16T04:36:07.374 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:36:07.487 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-cephfs 2024-01-16T04:36:08.340 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:39 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:36:09.817 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-cephfs-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:36:10.050 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:36:10.052 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-01-16T04:36:10.052 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:36:10.207 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rbd 2024-01-16T04:36:11.067 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:42 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:36:12.548 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rbd-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:36:12.778 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:36:12.780 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-01-16T04:36:12.780 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:36:12.888 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-fuse 2024-01-16T04:36:13.761 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:44 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:36:15.467 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:36:15.467 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:36:15.467 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-01-16T04:36:15.467 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:36:15.468 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:36:15.468 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse x86_64 2:16.2.14-449.ga70aebef.el8 ceph 88 k 2024-01-16T04:36:15.468 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:15.468 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:36:15.468 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:36:15.468 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-01-16T04:36:15.468 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:15.469 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 88 k 2024-01-16T04:36:15.469 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 240 k 2024-01-16T04:36:15.469 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:36:16.023 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-fuse-16.2.14-449.ga70aebef.el8.x86_64.rpm 160 kB/s | 88 kB 00:00 2024-01-16T04:36:16.023 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:36:16.024 INFO:teuthology.orchestra.run.smithi176.stdout:Total 159 kB/s | 88 kB 00:00 2024-01-16T04:36:16.024 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:36:16.034 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:36:16.034 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:36:16.086 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:36:16.089 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:36:16.202 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:36:16.438 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:36:16.729 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:36:17.410 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:36:17.410 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:17.410 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:36:17.410 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:36:17.410 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:17.411 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:36:17.598 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-mirror 2024-01-16T04:36:18.458 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:49 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:36:20.173 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:36:20.174 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:36:20.174 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-01-16T04:36:20.174 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:36:20.174 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:36:20.174 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror x86_64 2:16.2.14-449.ga70aebef.el8 ceph 3.7 M 2024-01-16T04:36:20.174 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:20.174 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:36:20.174 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:36:20.174 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-01-16T04:36:20.174 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:20.175 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 3.7 M 2024-01-16T04:36:20.175 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 15 M 2024-01-16T04:36:20.175 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:36:21.075 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-mirror-16.2.14-449.ga70aebef.el8.x86_64.rpm 4.1 MB/s | 3.7 MB 00:00 2024-01-16T04:36:21.076 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:36:21.076 INFO:teuthology.orchestra.run.smithi176.stdout:Total 4.1 MB/s | 3.7 MB 00:00 2024-01-16T04:36:21.076 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:36:21.088 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:36:21.088 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:36:21.152 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:36:21.155 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:36:21.598 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:36:21.788 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:36:22.208 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:36:22.915 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:36:22.915 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:22.916 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:36:22.916 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:36:22.916 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:22.916 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:36:23.116 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-nbd 2024-01-16T04:36:23.980 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:54 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:36:25.704 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:36:25.705 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:36:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-01-16T04:36:25.705 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:36:25.705 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:36:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd x86_64 2:16.2.14-449.ga70aebef.el8 ceph 175 k 2024-01-16T04:36:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:25.705 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:36:25.705 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:36:25.705 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-01-16T04:36:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:25.705 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 175 k 2024-01-16T04:36:25.705 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 523 k 2024-01-16T04:36:25.706 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:36:25.978 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-nbd-16.2.14-449.ga70aebef.el8.x86_64.rpm 643 kB/s | 175 kB 00:00 2024-01-16T04:36:25.978 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:36:25.978 INFO:teuthology.orchestra.run.smithi176.stdout:Total 640 kB/s | 175 kB 00:00 2024-01-16T04:36:25.979 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:36:25.989 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:36:25.989 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:36:26.039 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:36:26.042 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:36:26.160 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:36:26.336 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:36:26.625 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-nbd-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:36:27.358 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:36:27.358 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:27.358 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:36:27.358 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:36:27.358 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:27.358 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:36:27.577 DEBUG:teuthology.parallel:result is None 2024-01-16T04:36:27.578 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T04:36:27.843 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T04:36:27.867 INFO:teuthology.orchestra.run.smithi176.stdout:16.2.14-449.ga70aebef.el8 2024-01-16T04:36:27.867 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-449.ga70aebef.el8 2024-01-16T04:36:27.867 INFO:teuthology.task.install:The correct ceph version 16.2.14-449.ga70aebef is installed. 2024-01-16T04:36:27.869 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-16T04:36:27.869 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:27.869 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T04:36:27.936 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-16T04:36:27.937 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:27.937 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T04:36:28.004 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T04:36:28.071 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-16T04:36:28.071 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:28.071 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T04:36:28.137 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T04:36:28.204 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-16T04:36:28.204 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:28.204 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T04:36:28.271 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T04:36:28.338 INFO:teuthology.run_tasks:Running task ceph... 2024-01-16T04:36:28.392 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-16T04:36:28.393 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /var/log/ceph 2024-01-16T04:36:28.419 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-16T04:36:28.419 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T04:36:28.485 INFO:tasks.ceph:Creating extra log directories... 2024-01-16T04:36:28.486 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T04:36:28.559 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-16T04:36:28.560 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/kv/data', 'rgw crypt vault secret engine': 'kv', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-16T04:36:28.560 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517322', 'branch': 'wip-yuri5-testing-2024-01-11-1300-pacific', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7517322', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2024-01-11-1300-pacific'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/kv/data', 'rgw crypt vault secret engine': 'kv', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'}}, 'rgw': {'client.0': {'use-vault-role': 'client.0'}}, 'workunit': {'branch': 'wip-yuri5-testing-2024-01-11-1300-pacific', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9013, 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri5-testing-2024-01-11-1300-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'targets': {'smithi176.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMIJErr9dGrhWCSEySf7+DUoB2WqQZ+JCMm6HItnXIkSTf0om2vY9E3f8+9olOrKYtE9tuLk7Llog4RR4LbdBc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'vault': {'client.0': {'engine': 'kv', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/kv/data/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1', 'secret': 'a2V5MS5GcWVxKzhzTGNLaGtzQkg5NGVpb1FKcFpGb2c='}, {'path': 'my-key-2', 'secret': 'a2V5Mi5yNUNNMGFzMVdIUVZxcCt5NGVmVGlQQ1k4YWg='}]}}}, {'rgw': {'client.0': None}}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-pacific', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-15_22:05:22', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-16T04:36:28.560 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T04:36:28.610 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T04:36:28.677 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:28.678 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T04:36:28.735 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T04:36:28.735 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-01-16T04:36:28.793 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T04:36:28.793 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T04:36:28.793 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 161429 Links: 1 2024-01-16T04:36:28.793 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T04:36:28.793 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T04:36:28.793 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-01-16 04:36:21.964701300 +0000 2024-01-16T04:36:28.793 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-01-16 04:32:23.358801751 +0000 2024-01-16T04:36:28.793 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-01-16 04:32:23.358801751 +0000 2024-01-16T04:36:28.793 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-01-16T04:36:28.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T04:36:28.858 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-01-16T04:36:28.858 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-01-16T04:36:28.858 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000235391 s, 2.2 MB/s 2024-01-16T04:36:28.859 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T04:36:28.918 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-01-16T04:36:28.976 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T04:36:28.976 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T04:36:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 158688 Links: 1 2024-01-16T04:36:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T04:36:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T04:36:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-01-16 04:36:21.964701300 +0000 2024-01-16T04:36:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-01-16 04:32:23.697798637 +0000 2024-01-16T04:36:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-01-16 04:32:23.697798637 +0000 2024-01-16T04:36:28.976 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-01-16T04:36:28.977 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T04:36:29.041 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-01-16T04:36:29.042 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-01-16T04:36:29.042 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000237848 s, 2.2 MB/s 2024-01-16T04:36:29.043 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T04:36:29.102 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-01-16T04:36:29.160 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T04:36:29.160 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T04:36:29.160 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 165951 Links: 1 2024-01-16T04:36:29.160 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T04:36:29.160 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T04:36:29.160 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-01-16 04:36:21.964701300 +0000 2024-01-16T04:36:29.161 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-01-16 04:32:24.037795515 +0000 2024-01-16T04:36:29.161 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-01-16 04:32:24.037795515 +0000 2024-01-16T04:36:29.161 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-01-16T04:36:29.161 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T04:36:29.225 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-01-16T04:36:29.225 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-01-16T04:36:29.225 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000161433 s, 3.2 MB/s 2024-01-16T04:36:29.226 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T04:36:29.285 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-01-16T04:36:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T04:36:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T04:36:29.343 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 166051 Links: 1 2024-01-16T04:36:29.343 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T04:36:29.343 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T04:36:29.343 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-01-16 04:36:21.964701300 +0000 2024-01-16T04:36:29.343 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-01-16 04:32:24.384792328 +0000 2024-01-16T04:36:29.343 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-01-16 04:32:24.384792328 +0000 2024-01-16T04:36:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-01-16T04:36:29.344 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T04:36:29.409 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-01-16T04:36:29.409 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-01-16T04:36:29.409 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000225789 s, 2.3 MB/s 2024-01-16T04:36:29.410 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T04:36:29.469 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-16T04:36:29.469 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-16T04:36:29.469 INFO:tasks.ceph:Generating config... 2024-01-16T04:36:29.470 INFO:tasks.ceph:[client] debug rgw = 20 2024-01-16T04:36:29.470 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-01-16T04:36:29.470 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = vault 2024-01-16T04:36:29.470 INFO:tasks.ceph:[client] rgw crypt vault auth = token 2024-01-16T04:36:29.470 INFO:tasks.ceph:[client] rgw crypt vault prefix = /v1/kv/data 2024-01-16T04:36:29.470 INFO:tasks.ceph:[client] rgw crypt vault secret engine = kv 2024-01-16T04:36:29.470 INFO:tasks.ceph:[client] setgroup = ceph 2024-01-16T04:36:29.470 INFO:tasks.ceph:[client] setuser = ceph 2024-01-16T04:36:29.470 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-16T04:36:29.470 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-16T04:36:29.471 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-16T04:36:29.471 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-16T04:36:29.471 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-16T04:36:29.471 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-16T04:36:29.471 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-16T04:36:29.471 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-16T04:36:29.471 INFO:tasks.ceph:Setting up mon.a... 2024-01-16T04:36:29.471 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-16T04:36:29.627 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/ceph.keyring 2024-01-16T04:36:29.629 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-16T04:36:29.670 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T04:36:29.737 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.176')] 2024-01-16T04:36:29.737 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.176'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/kv/data', 'rgw crypt vault secret engine': 'kv', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2024-01-16T04:36:29.738 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:29.738 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T04:36:29.795 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.176 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:36:29.872 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:36:29.872 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:36:29.873 INFO:teuthology.orchestra.run.smithi176.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:36:29.873 INFO:teuthology.orchestra.run.smithi176.stdout:monmaptool: generated fsid 43305b7c-7b81-41aa-8897-fc8a960bca4f 2024-01-16T04:36:29.873 INFO:teuthology.orchestra.run.smithi176.stdout:epoch 0 2024-01-16T04:36:29.873 INFO:teuthology.orchestra.run.smithi176.stdout:fsid 43305b7c-7b81-41aa-8897-fc8a960bca4f 2024-01-16T04:36:29.873 INFO:teuthology.orchestra.run.smithi176.stdout:last_changed 2024-01-16T04:36:29.877268+0000 2024-01-16T04:36:29.873 INFO:teuthology.orchestra.run.smithi176.stdout:created 2024-01-16T04:36:29.877268+0000 2024-01-16T04:36:29.873 INFO:teuthology.orchestra.run.smithi176.stdout:min_mon_release 0 (unknown) 2024-01-16T04:36:29.873 INFO:teuthology.orchestra.run.smithi176.stdout:election_strategy: 1 2024-01-16T04:36:29.874 INFO:teuthology.orchestra.run.smithi176.stdout:0: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.a 2024-01-16T04:36:29.874 INFO:teuthology.orchestra.run.smithi176.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-01-16T04:36:29.875 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T04:36:29.932 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 43305b7c-7b81-41aa-8897-fc8a960bca4f... 2024-01-16T04:36:29.932 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T04:36:30.016 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2024-01-16T04:36:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2024-01-16T04:36:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T04:36:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2024-01-16T04:36:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2024-01-16T04:36:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2024-01-16T04:36:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2024-01-16T04:36:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: mon_allow_pool_size_one = true 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = 172.21.15.176 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = 43305b7c-7b81-41aa-8897-fc8a960bca4f 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2024-01-16T04:36:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2024-01-16T04:36:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2024-01-16T04:36:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2024-01-16T04:36:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2024-01-16T04:36:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2024-01-16T04:36:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2024-01-16T04:36:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2024-01-16T04:36:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: osd shutdown pgref assert = True 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2024-01-16T04:36:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2024-01-16T04:36:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2024-01-16T04:36:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: debug rgw = 20 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: rgw crypt require ssl = False 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: rgw crypt s3 kms backend = vault 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: rgw crypt vault auth = token 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: rgw crypt vault prefix = /v1/kv/data 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: rgw crypt vault secret engine = kv 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: setgroup = ceph 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: setuser = ceph 2024-01-16T04:36:30.024 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2024-01-16T04:36:30.026 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-16T04:36:30.026 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-16T04:36:30.118 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-16T04:36:30.118 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:30.118 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-16T04:36:30.135 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:30.135 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-16T04:36:30.193 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2024-01-16T04:36:30.193 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:30.193 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T04:36:30.193 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T04:36:30.269 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:30.270 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:36:30.327 INFO:tasks.ceph:Setting up mon nodes... 2024-01-16T04:36:30.327 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-16T04:36:30.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T04:36:30.416 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T04:36:30.418 INFO:tasks.ceph:Setting up mds nodes... 2024-01-16T04:36:30.418 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-16T04:36:30.419 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-16T04:36:30.457 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-16T04:36:30.469 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-16T04:36:30.469 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-16T04:36:30.470 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-16T04:36:30.536 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-16T04:36:30.536 INFO:tasks.ceph:role: osd.0 2024-01-16T04:36:30.537 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com 2024-01-16T04:36:30.537 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T04:36:30.611 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T04:36:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T04:36:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T04:36:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T04:36:30.611 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T04:36:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-01-16T04:36:30.611 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T04:36:30.611 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T04:36:30.611 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T04:36:30.612 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T04:36:31.046 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-01-16T04:36:31.048 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-01-16T04:36:31.048 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-16T04:36:31.236 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-16T04:36:31.269 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-16T04:36:31.336 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-16T04:36:31.336 INFO:tasks.ceph:role: osd.1 2024-01-16T04:36:31.337 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com 2024-01-16T04:36:31.337 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T04:36:31.410 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T04:36:31.411 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T04:36:31.411 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T04:36:31.411 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T04:36:31.411 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T04:36:31.411 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-01-16T04:36:31.411 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T04:36:31.411 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T04:36:31.411 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T04:36:31.411 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T04:36:31.834 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-01-16T04:36:31.836 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-01-16T04:36:31.836 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-16T04:36:31.946 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-16T04:36:31.979 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-16T04:36:32.047 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-16T04:36:32.047 INFO:tasks.ceph:role: osd.2 2024-01-16T04:36:32.047 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com 2024-01-16T04:36:32.048 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T04:36:32.122 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T04:36:32.122 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T04:36:32.123 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T04:36:32.123 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T04:36:32.123 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T04:36:32.123 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-01-16T04:36:32.123 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T04:36:32.123 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T04:36:32.123 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T04:36:32.123 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T04:36:32.570 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-01-16T04:36:32.572 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-01-16T04:36:32.572 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-16T04:36:32.750 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-16T04:36:32.783 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:36:32.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:32.875+0000 7f1f5713c380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-16T04:36:32.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:32.876+0000 7f1f5713c380 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-16T04:36:32.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:32.876+0000 7f1f5713c380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T04:36:32.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:32.876+0000 7f1f5713c380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T04:36:32.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:32.876+0000 7f1f5713c380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T04:36:32.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:32.884+0000 7f1f5713c380 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-16T04:36:34.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-16T04:36:34.993 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:36:35.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:35.084+0000 7ff3c0596380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-16T04:36:35.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:35.084+0000 7ff3c0596380 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-16T04:36:35.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:35.084+0000 7ff3c0596380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T04:36:35.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:35.084+0000 7ff3c0596380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T04:36:35.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:35.084+0000 7ff3c0596380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T04:36:35.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:35.084+0000 7ff3c0596380 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-16T04:36:37.158 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-16T04:36:37.190 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:36:37.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:37.280+0000 7fa8c6df4380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-16T04:36:37.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:37.280+0000 7fa8c6df4380 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-16T04:36:37.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:37.280+0000 7fa8c6df4380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T04:36:37.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:37.280+0000 7fa8c6df4380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T04:36:37.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:37.280+0000 7fa8c6df4380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T04:36:37.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:36:37.281+0000 7fa8c6df4380 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-16T04:36:39.350 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-16T04:36:39.383 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-16T04:36:39.383 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:39.383 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-16T04:36:39.453 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:39.453 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-16T04:36:39.522 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:39.522 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-16T04:36:39.590 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:39.590 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-16T04:36:39.658 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:39.658 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-16T04:36:39.716 INFO:tasks.ceph:Adding keys to all mons... 2024-01-16T04:36:39.717 DEBUG:teuthology.orchestra.run.smithi176:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T04:36:39.782 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.x] 2024-01-16T04:36:39.782 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDOB6Zlvr8UGRAAVn/tcMcwmB6+rxlVpXUySQ== 2024-01-16T04:36:39.782 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.0] 2024-01-16T04:36:39.783 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDQB6ZlnBs/NBAA5J0Qz1rYWcEMJymikl5ziA== 2024-01-16T04:36:39.783 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.1] 2024-01-16T04:36:39.783 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDTB6ZlD98OBRAAT5WY+SrXLwiJiTUVBQv/jg== 2024-01-16T04:36:39.783 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.2] 2024-01-16T04:36:39.783 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDVB6Zleh3FEBAAwP2KZduKU0nl4sbD757N2A== 2024-01-16T04:36:39.783 INFO:teuthology.orchestra.run.smithi176.stdout:[client.0] 2024-01-16T04:36:39.783 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDOB6ZlmIKIGxAABmBea/UA8w+o+uzgBlVImg== 2024-01-16T04:36:39.784 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T04:36:39.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T04:36:39.920 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T04:36:39.968 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T04:36:40.019 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T04:36:40.069 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-16T04:36:40.069 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-16T04:36:40.095 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T04:36:40.189 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-16T04:36:40.216 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:36:40.273 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-16T04:36:40.273 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-16T04:36:40.273 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-16T04:36:40.316 INFO:tasks.ceph.mon.a:Started 2024-01-16T04:36:40.316 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-16T04:36:40.316 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-16T04:36:40.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-16T04:36:40.318 INFO:tasks.ceph.mgr.x:Started 2024-01-16T04:36:40.318 INFO:tasks.ceph:Setting crush tunables to default 2024-01-16T04:36:40.318 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd crush tunables default 2024-01-16T04:36:40.533 INFO:teuthology.orchestra.run.smithi176.stderr:adjusted tunables profile to default 2024-01-16T04:36:40.544 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-16T04:36:40.544 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:40.544 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-16T04:36:40.570 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:40.570 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-16T04:36:40.637 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:36:40.637 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-16T04:36:40.703 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 847d61e2-90f6-4b18-9185-770c7d2aa786 0 2024-01-16T04:36:40.740 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:40.744+0000 7f4b69c5a680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T04:36:40.812 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:40.816+0000 7f4b69c5a680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T04:36:40.944 INFO:teuthology.orchestra.run.smithi176.stdout:0 2024-01-16T04:36:40.956 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new d5421cb7-0f31-4225-8ae7-ab6b68e16c62 1 2024-01-16T04:36:41.158 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-01-16T04:36:41.170 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 9e5cf429-dd1d-41ee-a3d0-36013414c1fe 2 2024-01-16T04:36:41.261 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:41.264+0000 7f4b69c5a680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T04:36:41.373 INFO:teuthology.orchestra.run.smithi176.stdout:2 2024-01-16T04:36:41.385 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-16T04:36:41.385 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-16T04:36:41.387 INFO:tasks.ceph.osd.0:Started 2024-01-16T04:36:41.387 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-16T04:36:41.387 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-16T04:36:41.391 INFO:tasks.ceph.osd.1:Started 2024-01-16T04:36:41.391 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-16T04:36:41.391 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-16T04:36:41.393 INFO:tasks.ceph.osd.2:Started 2024-01-16T04:36:41.394 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T04:36:41.607 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:41.607 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":5,"fsid":"43305b7c-7b81-41aa-8897-fc8a960bca4f","created":"2024-01-16T04:36:40.444462+0000","modified":"2024-01-16T04:36:41.376078+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T04:36:41.376078+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"847d61e2-90f6-4b18-9185-770c7d2aa786","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d5421cb7-0f31-4225-8ae7-ab6b68e16c62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e5cf429-dd1d-41ee-a3d0-36013414c1fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:36:41.620 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-16T04:36:41.621 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-16T04:36:41.923 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T04:36:42.031 INFO:tasks.ceph.osd.0.smithi176.stderr:2024-01-16T04:36:42.034+0000 7f2ec705c380 -1 Falling back to public interface 2024-01-16T04:36:42.036 INFO:tasks.ceph.osd.1.smithi176.stderr:2024-01-16T04:36:42.039+0000 7f2faed2b380 -1 Falling back to public interface 2024-01-16T04:36:42.036 INFO:tasks.ceph.osd.2.smithi176.stderr:2024-01-16T04:36:42.039+0000 7fa617ef3380 -1 Falling back to public interface 2024-01-16T04:36:42.049 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:42.053+0000 7f4b69c5a680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T04:36:42.119 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:42.123+0000 7f4b69c5a680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T04:36:42.127 INFO:teuthology.misc.health.smithi176.stdout: 2024-01-16T04:36:42.127 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":5,"fsid":"43305b7c-7b81-41aa-8897-fc8a960bca4f","created":"2024-01-16T04:36:40.444462+0000","modified":"2024-01-16T04:36:41.376078+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T04:36:41.376078+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"847d61e2-90f6-4b18-9185-770c7d2aa786","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d5421cb7-0f31-4225-8ae7-ab6b68e16c62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e5cf429-dd1d-41ee-a3d0-36013414c1fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:36:42.138 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-01-16T04:36:42.311 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:42.314+0000 7f4b69c5a680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T04:36:42.747 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:42.751+0000 7f4b69c5a680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T04:36:42.849 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:42.852+0000 7f4b69c5a680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T04:36:43.002 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:43.006+0000 7f4b69c5a680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T04:36:43.079 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:43.082+0000 7f4b69c5a680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T04:36:43.149 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:43.152+0000 7f4b69c5a680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T04:36:43.405 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:43.408+0000 7f4b69c5a680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T04:36:43.616 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:43.619+0000 7f4b69c5a680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T04:36:44.279 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:44.282+0000 7f4b69c5a680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T04:36:44.347 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:44.350+0000 7f4b69c5a680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T04:36:44.425 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:44.428+0000 7f4b69c5a680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T04:36:44.575 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:44.579+0000 7f4b69c5a680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T04:36:44.646 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:44.649+0000 7f4b69c5a680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T04:36:44.761 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:44.764+0000 7f4b69c5a680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T04:36:44.775 INFO:tasks.ceph.osd.0.smithi176.stderr:2024-01-16T04:36:44.778+0000 7f2ec705c380 -1 osd.0 0 log_to_monitors {default=true} 2024-01-16T04:36:44.787 INFO:tasks.ceph.osd.2.smithi176.stderr:2024-01-16T04:36:44.790+0000 7fa617ef3380 -1 osd.2 0 log_to_monitors {default=true} 2024-01-16T04:36:44.796 INFO:tasks.ceph.osd.1.smithi176.stderr:2024-01-16T04:36:44.800+0000 7f2faed2b380 -1 osd.1 0 log_to_monitors {default=true} 2024-01-16T04:36:44.879 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:44.882+0000 7f4b69c5a680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T04:36:45.199 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:45.203+0000 7f4b69c5a680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T04:36:45.271 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:45.274+0000 7f4b69c5a680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T04:36:45.838 INFO:tasks.ceph.osd.0.smithi176.stderr:2024-01-16T04:36:45.841+0000 7f2ebd9fd700 -1 osd.0 0 waiting for initial osdmap 2024-01-16T04:36:45.839 INFO:tasks.ceph.osd.2.smithi176.stderr:2024-01-16T04:36:45.841+0000 7fa60e894700 -1 osd.2 0 waiting for initial osdmap 2024-01-16T04:36:45.839 INFO:tasks.ceph.osd.1.smithi176.stderr:2024-01-16T04:36:45.842+0000 7f2fa56cc700 -1 osd.1 0 waiting for initial osdmap 2024-01-16T04:36:45.843 INFO:tasks.ceph.osd.0.smithi176.stderr:2024-01-16T04:36:45.846+0000 7f2eba179700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T04:36:45.843 INFO:tasks.ceph.osd.2.smithi176.stderr:2024-01-16T04:36:45.846+0000 7fa60b010700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T04:36:45.843 INFO:tasks.ceph.osd.1.smithi176.stderr:2024-01-16T04:36:45.846+0000 7f2fa1e48700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T04:36:46.290 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-01-16T04:36:46.294+0000 7f4b1f173700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-01-16T04:36:48.441 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T04:36:48.730 INFO:teuthology.misc.health.smithi176.stdout: 2024-01-16T04:36:48.731 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":9,"fsid":"43305b7c-7b81-41aa-8897-fc8a960bca4f","created":"2024-01-16T04:36:40.444462+0000","modified":"2024-01-16T04:36:47.847361+0000","last_up_change":"2024-01-16T04:36:46.843164+0000","last_in_change":"2024-01-16T04:36:41.376078+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-16T04:36:46.881958+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"847d61e2-90f6-4b18-9185-770c7d2aa786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":40345},{"type":"v1","addr":"172.21.15.176:6801","nonce":40345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":40345},{"type":"v1","addr":"172.21.15.176:6803","nonce":40345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":40345},{"type":"v1","addr":"172.21.15.176:6807","nonce":40345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":40345},{"type":"v1","addr":"172.21.15.176:6805","nonce":40345}]},"public_addr":"172.21.15.176:6801/40345","cluster_addr":"172.21.15.176:6803/40345","heartbeat_back_addr":"172.21.15.176:6807/40345","heartbeat_front_addr":"172.21.15.176:6805/40345","state":["exists","up"]},{"osd":1,"uuid":"d5421cb7-0f31-4225-8ae7-ab6b68e16c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":40360},{"type":"v1","addr":"172.21.15.176:6810","nonce":40360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":40360},{"type":"v1","addr":"172.21.15.176:6814","nonce":40360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":40360},{"type":"v1","addr":"172.21.15.176:6822","nonce":40360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":40360},{"type":"v1","addr":"172.21.15.176:6818","nonce":40360}]},"public_addr":"172.21.15.176:6810/40360","cluster_addr":"172.21.15.176:6814/40360","heartbeat_back_addr":"172.21.15.176:6822/40360","heartbeat_front_addr":"172.21.15.176:6818/40360","state":["exists","up"]},{"osd":2,"uuid":"9e5cf429-dd1d-41ee-a3d0-36013414c1fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":40361},{"type":"v1","addr":"172.21.15.176:6811","nonce":40361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":40361},{"type":"v1","addr":"172.21.15.176:6815","nonce":40361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":40361},{"type":"v1","addr":"172.21.15.176:6823","nonce":40361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":40361},{"type":"v1","addr":"172.21.15.176:6819","nonce":40361}]},"public_addr":"172.21.15.176:6811/40361","cluster_addr":"172.21.15.176:6815/40361","heartbeat_back_addr":"172.21.15.176:6823/40361","heartbeat_front_addr":"172.21.15.176:6819/40361","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:36:45.817427+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:36:45.844338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:36:45.794505+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:36:48.742 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-01-16T04:36:48.743 INFO:tasks.ceph:Creating RBD pool 2024-01-16T04:36:48.743 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-16T04:36:49.863 INFO:teuthology.orchestra.run.smithi176.stderr:pool 'rbd' created 2024-01-16T04:36:49.874 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-16T04:36:50.870 INFO:teuthology.orchestra.run.smithi176.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-16T04:36:50.883 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-16T04:36:50.884 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-16T04:36:50.884 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-16T04:36:50.885 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-16T04:36:50.885 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-16T04:36:51.217 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:51.229 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":39937},{"type":"v1","addr":"172.21.15.176:6825","nonce":39937}]},"active_addr":"172.21.15.176:6825/39937","active_change":"2024-01-16T04:36:45.287508+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":3705910551}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":1688386780}]}]} 2024-01-16T04:36:51.231 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-16T04:36:51.231 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-16T04:36:51.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T04:36:51.518 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:51.518 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":12,"fsid":"43305b7c-7b81-41aa-8897-fc8a960bca4f","created":"2024-01-16T04:36:40.444462+0000","modified":"2024-01-16T04:36:50.859124+0000","last_up_change":"2024-01-16T04:36:46.843164+0000","last_in_change":"2024-01-16T04:36:41.376078+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-16T04:36:46.881958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-16T04:36:49.062116+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"847d61e2-90f6-4b18-9185-770c7d2aa786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":40345},{"type":"v1","addr":"172.21.15.176:6801","nonce":40345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":40345},{"type":"v1","addr":"172.21.15.176:6803","nonce":40345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":40345},{"type":"v1","addr":"172.21.15.176:6807","nonce":40345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":40345},{"type":"v1","addr":"172.21.15.176:6805","nonce":40345}]},"public_addr":"172.21.15.176:6801/40345","cluster_addr":"172.21.15.176:6803/40345","heartbeat_back_addr":"172.21.15.176:6807/40345","heartbeat_front_addr":"172.21.15.176:6805/40345","state":["exists","up"]},{"osd":1,"uuid":"d5421cb7-0f31-4225-8ae7-ab6b68e16c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":40360},{"type":"v1","addr":"172.21.15.176:6810","nonce":40360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":40360},{"type":"v1","addr":"172.21.15.176:6814","nonce":40360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":40360},{"type":"v1","addr":"172.21.15.176:6822","nonce":40360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":40360},{"type":"v1","addr":"172.21.15.176:6818","nonce":40360}]},"public_addr":"172.21.15.176:6810/40360","cluster_addr":"172.21.15.176:6814/40360","heartbeat_back_addr":"172.21.15.176:6822/40360","heartbeat_front_addr":"172.21.15.176:6818/40360","state":["exists","up"]},{"osd":2,"uuid":"9e5cf429-dd1d-41ee-a3d0-36013414c1fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":40361},{"type":"v1","addr":"172.21.15.176:6811","nonce":40361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":40361},{"type":"v1","addr":"172.21.15.176:6815","nonce":40361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":40361},{"type":"v1","addr":"172.21.15.176:6823","nonce":40361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":40361},{"type":"v1","addr":"172.21.15.176:6819","nonce":40361}]},"public_addr":"172.21.15.176:6811/40361","cluster_addr":"172.21.15.176:6815/40361","heartbeat_back_addr":"172.21.15.176:6823/40361","heartbeat_front_addr":"172.21.15.176:6819/40361","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:36:45.817427+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:36:45.844338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:36:45.794505+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:36:51.532 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-16T04:36:51.532 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T04:36:51.821 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:51.821 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":12,"fsid":"43305b7c-7b81-41aa-8897-fc8a960bca4f","created":"2024-01-16T04:36:40.444462+0000","modified":"2024-01-16T04:36:50.859124+0000","last_up_change":"2024-01-16T04:36:46.843164+0000","last_in_change":"2024-01-16T04:36:41.376078+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-16T04:36:46.881958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-16T04:36:49.062116+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"847d61e2-90f6-4b18-9185-770c7d2aa786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":40345},{"type":"v1","addr":"172.21.15.176:6801","nonce":40345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":40345},{"type":"v1","addr":"172.21.15.176:6803","nonce":40345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":40345},{"type":"v1","addr":"172.21.15.176:6807","nonce":40345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":40345},{"type":"v1","addr":"172.21.15.176:6805","nonce":40345}]},"public_addr":"172.21.15.176:6801/40345","cluster_addr":"172.21.15.176:6803/40345","heartbeat_back_addr":"172.21.15.176:6807/40345","heartbeat_front_addr":"172.21.15.176:6805/40345","state":["exists","up"]},{"osd":1,"uuid":"d5421cb7-0f31-4225-8ae7-ab6b68e16c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":40360},{"type":"v1","addr":"172.21.15.176:6810","nonce":40360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":40360},{"type":"v1","addr":"172.21.15.176:6814","nonce":40360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":40360},{"type":"v1","addr":"172.21.15.176:6822","nonce":40360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":40360},{"type":"v1","addr":"172.21.15.176:6818","nonce":40360}]},"public_addr":"172.21.15.176:6810/40360","cluster_addr":"172.21.15.176:6814/40360","heartbeat_back_addr":"172.21.15.176:6822/40360","heartbeat_front_addr":"172.21.15.176:6818/40360","state":["exists","up"]},{"osd":2,"uuid":"9e5cf429-dd1d-41ee-a3d0-36013414c1fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":40361},{"type":"v1","addr":"172.21.15.176:6811","nonce":40361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":40361},{"type":"v1","addr":"172.21.15.176:6815","nonce":40361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":40361},{"type":"v1","addr":"172.21.15.176:6823","nonce":40361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":40361},{"type":"v1","addr":"172.21.15.176:6819","nonce":40361}]},"public_addr":"172.21.15.176:6811/40361","cluster_addr":"172.21.15.176:6815/40361","heartbeat_back_addr":"172.21.15.176:6823/40361","heartbeat_front_addr":"172.21.15.176:6819/40361","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:36:45.817427+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:36:45.844338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:36:45.794505+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:36:51.834 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-16T04:36:51.979 INFO:teuthology.orchestra.run.smithi176.stdout:34359738371 2024-01-16T04:36:51.979 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-16T04:36:52.123 INFO:teuthology.orchestra.run.smithi176.stdout:34359738371 2024-01-16T04:36:52.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-16T04:36:52.270 INFO:teuthology.orchestra.run.smithi176.stdout:34359738370 2024-01-16T04:36:52.270 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T04:36:52.553 INFO:teuthology.orchestra.run.smithi176.stdout:34359738370 2024-01-16T04:36:52.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2024-01-16T04:36:53.567 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T04:36:53.860 INFO:teuthology.orchestra.run.smithi176.stdout:34359738370 2024-01-16T04:36:53.873 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2024-01-16T04:36:54.874 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T04:36:55.172 INFO:teuthology.orchestra.run.smithi176.stdout:34359738372 2024-01-16T04:36:55.186 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738372 for osd.0 2024-01-16T04:36:55.186 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T04:36:55.473 INFO:teuthology.orchestra.run.smithi176.stdout:34359738372 2024-01-16T04:36:55.486 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738372 for osd.1 2024-01-16T04:36:55.486 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T04:36:55.770 INFO:teuthology.orchestra.run.smithi176.stdout:34359738371 2024-01-16T04:36:55.783 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738371 for osd.2 2024-01-16T04:36:55.783 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T04:36:55.783 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T04:36:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:56.064 INFO:teuthology.orchestra.run.smithi176.stderr:dumped all 2024-01-16T04:36:56.075 INFO:teuthology.orchestra.run.smithi176.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-01-16T04:36:55.292768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":891600,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":313681200,"statfs":{"total":322122547200,"available":321209548800,"internally_reserved":0,"allocated":319488,"data_stored":86889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.873409+0000","last_change":"2024-01-16T04:36:50.872527+0000","last_active":"2024-01-16T04:36:50.873409+0000","last_peered":"2024-01-16T04:36:50.873409+0000","last_clean":"2024-01-16T04:36:50.873409+0000","last_became_active":"2024-01-16T04:36:50.872227+0000","last_became_peered":"2024-01-16T04:36:50.872227+0000","last_unstale":"2024-01-16T04:36:50.873409+0000","last_undegraded":"2024-01-16T04:36:50.873409+0000","last_fullsized":"2024-01-16T04:36:50.873409+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.873413+0000","last_change":"2024-01-16T04:36:50.872120+0000","last_active":"2024-01-16T04:36:50.873413+0000","last_peered":"2024-01-16T04:36:50.873413+0000","last_clean":"2024-01-16T04:36:50.873413+0000","last_became_active":"2024-01-16T04:36:50.871480+0000","last_became_peered":"2024-01-16T04:36:50.871480+0000","last_unstale":"2024-01-16T04:36:50.873413+0000","last_undegraded":"2024-01-16T04:36:50.873413+0000","last_fullsized":"2024-01-16T04:36:50.873413+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.873402+0000","last_change":"2024-01-16T04:36:50.872305+0000","last_active":"2024-01-16T04:36:50.873402+0000","last_peered":"2024-01-16T04:36:50.873402+0000","last_clean":"2024-01-16T04:36:50.873402+0000","last_became_active":"2024-01-16T04:36:50.871502+0000","last_became_peered":"2024-01-16T04:36:50.871502+0000","last_unstale":"2024-01-16T04:36:50.873402+0000","last_undegraded":"2024-01-16T04:36:50.873402+0000","last_fullsized":"2024-01-16T04:36:50.873402+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.872169+0000","last_change":"2024-01-16T04:36:50.872169+0000","last_active":"2024-01-16T04:36:50.872169+0000","last_peered":"2024-01-16T04:36:50.872169+0000","last_clean":"2024-01-16T04:36:50.872169+0000","last_became_active":"2024-01-16T04:36:50.871819+0000","last_became_peered":"2024-01-16T04:36:50.871819+0000","last_unstale":"2024-01-16T04:36:50.872169+0000","last_undegraded":"2024-01-16T04:36:50.872169+0000","last_fullsized":"2024-01-16T04:36:50.872169+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.868775+0000","last_change":"2024-01-16T04:36:48.856941+0000","last_active":"2024-01-16T04:36:50.868775+0000","last_peered":"2024-01-16T04:36:50.868775+0000","last_clean":"2024-01-16T04:36:50.868775+0000","last_became_active":"2024-01-16T04:36:48.856661+0000","last_became_peered":"2024-01-16T04:36:48.856661+0000","last_unstale":"2024-01-16T04:36:50.868775+0000","last_undegraded":"2024-01-16T04:36:50.868775+0000","last_fullsized":"2024-01-16T04:36:50.868775+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:47.847361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:47.847361+0000","last_clean_scrub_stamp":"2024-01-16T04:36:47.847361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.873433+0000","last_change":"2024-01-16T04:36:50.872649+0000","last_active":"2024-01-16T04:36:50.873433+0000","last_peered":"2024-01-16T04:36:50.873433+0000","last_clean":"2024-01-16T04:36:50.873433+0000","last_became_active":"2024-01-16T04:36:50.872414+0000","last_became_peered":"2024-01-16T04:36:50.872414+0000","last_unstale":"2024-01-16T04:36:50.873433+0000","last_undegraded":"2024-01-16T04:36:50.873433+0000","last_fullsized":"2024-01-16T04:36:50.873433+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.872056+0000","last_change":"2024-01-16T04:36:50.872056+0000","last_active":"2024-01-16T04:36:50.872056+0000","last_peered":"2024-01-16T04:36:50.872056+0000","last_clean":"2024-01-16T04:36:50.872056+0000","last_became_active":"2024-01-16T04:36:50.871735+0000","last_became_peered":"2024-01-16T04:36:50.871735+0000","last_unstale":"2024-01-16T04:36:50.872056+0000","last_undegraded":"2024-01-16T04:36:50.872056+0000","last_fullsized":"2024-01-16T04:36:50.872056+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.873858+0000","last_change":"2024-01-16T04:36:50.873858+0000","last_active":"2024-01-16T04:36:50.873858+0000","last_peered":"2024-01-16T04:36:50.873858+0000","last_clean":"2024-01-16T04:36:50.873858+0000","last_became_active":"2024-01-16T04:36:50.873566+0000","last_became_peered":"2024-01-16T04:36:50.873566+0000","last_unstale":"2024-01-16T04:36:50.873858+0000","last_undegraded":"2024-01-16T04:36:50.873858+0000","last_fullsized":"2024-01-16T04:36:50.873858+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.873446+0000","last_change":"2024-01-16T04:36:50.872368+0000","last_active":"2024-01-16T04:36:50.873446+0000","last_peered":"2024-01-16T04:36:50.873446+0000","last_clean":"2024-01-16T04:36:50.873446+0000","last_became_active":"2024-01-16T04:36:50.871590+0000","last_became_peered":"2024-01-16T04:36:50.871590+0000","last_unstale":"2024-01-16T04:36:50.873446+0000","last_undegraded":"2024-01-16T04:36:50.873446+0000","last_fullsized":"2024-01-16T04:36:50.873446+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T04:36:56.076 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T04:36:56.357 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:56.359 INFO:teuthology.orchestra.run.smithi176.stderr:dumped all 2024-01-16T04:36:56.369 INFO:teuthology.orchestra.run.smithi176.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-01-16T04:36:55.292768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":891600,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":313681200,"statfs":{"total":322122547200,"available":321209548800,"internally_reserved":0,"allocated":319488,"data_stored":86889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.873409+0000","last_change":"2024-01-16T04:36:50.872527+0000","last_active":"2024-01-16T04:36:50.873409+0000","last_peered":"2024-01-16T04:36:50.873409+0000","last_clean":"2024-01-16T04:36:50.873409+0000","last_became_active":"2024-01-16T04:36:50.872227+0000","last_became_peered":"2024-01-16T04:36:50.872227+0000","last_unstale":"2024-01-16T04:36:50.873409+0000","last_undegraded":"2024-01-16T04:36:50.873409+0000","last_fullsized":"2024-01-16T04:36:50.873409+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.873413+0000","last_change":"2024-01-16T04:36:50.872120+0000","last_active":"2024-01-16T04:36:50.873413+0000","last_peered":"2024-01-16T04:36:50.873413+0000","last_clean":"2024-01-16T04:36:50.873413+0000","last_became_active":"2024-01-16T04:36:50.871480+0000","last_became_peered":"2024-01-16T04:36:50.871480+0000","last_unstale":"2024-01-16T04:36:50.873413+0000","last_undegraded":"2024-01-16T04:36:50.873413+0000","last_fullsized":"2024-01-16T04:36:50.873413+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.873402+0000","last_change":"2024-01-16T04:36:50.872305+0000","last_active":"2024-01-16T04:36:50.873402+0000","last_peered":"2024-01-16T04:36:50.873402+0000","last_clean":"2024-01-16T04:36:50.873402+0000","last_became_active":"2024-01-16T04:36:50.871502+0000","last_became_peered":"2024-01-16T04:36:50.871502+0000","last_unstale":"2024-01-16T04:36:50.873402+0000","last_undegraded":"2024-01-16T04:36:50.873402+0000","last_fullsized":"2024-01-16T04:36:50.873402+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.872169+0000","last_change":"2024-01-16T04:36:50.872169+0000","last_active":"2024-01-16T04:36:50.872169+0000","last_peered":"2024-01-16T04:36:50.872169+0000","last_clean":"2024-01-16T04:36:50.872169+0000","last_became_active":"2024-01-16T04:36:50.871819+0000","last_became_peered":"2024-01-16T04:36:50.871819+0000","last_unstale":"2024-01-16T04:36:50.872169+0000","last_undegraded":"2024-01-16T04:36:50.872169+0000","last_fullsized":"2024-01-16T04:36:50.872169+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.868775+0000","last_change":"2024-01-16T04:36:48.856941+0000","last_active":"2024-01-16T04:36:50.868775+0000","last_peered":"2024-01-16T04:36:50.868775+0000","last_clean":"2024-01-16T04:36:50.868775+0000","last_became_active":"2024-01-16T04:36:48.856661+0000","last_became_peered":"2024-01-16T04:36:48.856661+0000","last_unstale":"2024-01-16T04:36:50.868775+0000","last_undegraded":"2024-01-16T04:36:50.868775+0000","last_fullsized":"2024-01-16T04:36:50.868775+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:47.847361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:47.847361+0000","last_clean_scrub_stamp":"2024-01-16T04:36:47.847361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.873433+0000","last_change":"2024-01-16T04:36:50.872649+0000","last_active":"2024-01-16T04:36:50.873433+0000","last_peered":"2024-01-16T04:36:50.873433+0000","last_clean":"2024-01-16T04:36:50.873433+0000","last_became_active":"2024-01-16T04:36:50.872414+0000","last_became_peered":"2024-01-16T04:36:50.872414+0000","last_unstale":"2024-01-16T04:36:50.873433+0000","last_undegraded":"2024-01-16T04:36:50.873433+0000","last_fullsized":"2024-01-16T04:36:50.873433+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.872056+0000","last_change":"2024-01-16T04:36:50.872056+0000","last_active":"2024-01-16T04:36:50.872056+0000","last_peered":"2024-01-16T04:36:50.872056+0000","last_clean":"2024-01-16T04:36:50.872056+0000","last_became_active":"2024-01-16T04:36:50.871735+0000","last_became_peered":"2024-01-16T04:36:50.871735+0000","last_unstale":"2024-01-16T04:36:50.872056+0000","last_undegraded":"2024-01-16T04:36:50.872056+0000","last_fullsized":"2024-01-16T04:36:50.872056+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.873858+0000","last_change":"2024-01-16T04:36:50.873858+0000","last_active":"2024-01-16T04:36:50.873858+0000","last_peered":"2024-01-16T04:36:50.873858+0000","last_clean":"2024-01-16T04:36:50.873858+0000","last_became_active":"2024-01-16T04:36:50.873566+0000","last_became_peered":"2024-01-16T04:36:50.873566+0000","last_unstale":"2024-01-16T04:36:50.873858+0000","last_undegraded":"2024-01-16T04:36:50.873858+0000","last_fullsized":"2024-01-16T04:36:50.873858+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:36:50.873446+0000","last_change":"2024-01-16T04:36:50.872368+0000","last_active":"2024-01-16T04:36:50.873446+0000","last_peered":"2024-01-16T04:36:50.873446+0000","last_clean":"2024-01-16T04:36:50.873446+0000","last_became_active":"2024-01-16T04:36:50.871590+0000","last_became_peered":"2024-01-16T04:36:50.871590+0000","last_unstale":"2024-01-16T04:36:50.873446+0000","last_undegraded":"2024-01-16T04:36:50.873446+0000","last_fullsized":"2024-01-16T04:36:50.873446+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:36:49.853055+0000","last_clean_scrub_stamp":"2024-01-16T04:36:49.853055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":28963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T04:36:56.370 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-16T04:36:56.371 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-16T04:36:56.371 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-16T04:36:56.371 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-16T04:36:56.689 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:36:56.702 INFO:teuthology.orchestra.run.smithi176.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-16T04:36:56.702 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-16T04:36:56.703 INFO:teuthology.run_tasks:Running task vault... 2024-01-16T04:36:56.732 DEBUG:tasks.vault:Vault config is {'client.0': {'engine': 'kv', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/kv/data/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1', 'secret': 'a2V5MS5GcWVxKzhzTGNLaGtzQkg5NGVpb1FKcFpGb2c='}, {'path': 'my-key-2', 'secret': 'a2V5Mi5yNUNNMGFzMVdIUVZxcCt5NGVmVGlQQ1k4YWg='}]}} 2024-01-16T04:36:56.732 INFO:tasks.vault:Downloading Vault... 2024-01-16T04:36:56.733 INFO:tasks.vault:Downloading Vault... 2024-01-16T04:36:56.733 DEBUG:teuthology.orchestra.run.smithi176:> curl -L https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip -o /home/ubuntu/cephtest/vault.zip 2024-01-16T04:36:56.767 INFO:teuthology.orchestra.run.smithi176.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:36:56.768 INFO:teuthology.orchestra.run.smithi176.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:36:57.855 INFO:teuthology.orchestra.run.smithi176.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 74 42.6M 74 31.7M 0 0 38.7M 0 0:00:01 --:--:-- 0:00:01 38.7M 100 42.6M 100 42.6M 0 0 39.1M 0 0:00:01 0:00:01 --:--:-- 39.1M 2024-01-16T04:36:57.856 INFO:tasks.vault:Verifying SHA256 signature... 2024-01-16T04:36:57.856 DEBUG:teuthology.orchestra.run.smithi176:> echo '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 /home/ubuntu/cephtest/vault.zip' | sha256sum --check --status 2024-01-16T04:36:58.028 INFO:tasks.vault:Extracting vault... 2024-01-16T04:36:58.028 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p /home/ubuntu/cephtest/vault 2024-01-16T04:36:58.046 DEBUG:teuthology.orchestra.run.smithi176:> unzip /home/ubuntu/cephtest/vault.zip -d /home/ubuntu/cephtest/vault 2024-01-16T04:36:58.107 INFO:teuthology.orchestra.run.smithi176.stdout:Archive: /home/ubuntu/cephtest/vault.zip 2024-01-16T04:36:58.862 INFO:teuthology.orchestra.run.smithi176.stdout: inflating: /home/ubuntu/cephtest/vault/vault 2024-01-16T04:36:58.862 INFO:tasks.vault:Starting Vault listening on 0.0.0.0:8200 ... 2024-01-16T04:36:58.862 INFO:tasks.vault.client.0:Restarting daemon 2024-01-16T04:36:58.863 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/vault && exec bash -c 'chmod +x /home/ubuntu/cephtest/vault/vault && /home/ubuntu/cephtest/vault/vault server -dev -dev-listen-address=0.0.0.0:8200 -dev-no-store-token -dev-root-token-id=test_root_token' & { read; kill %1; }) 2024-01-16T04:36:58.905 INFO:tasks.vault.client.0:Started 2024-01-16T04:36:58.954 INFO:tasks.vault.client.0.smithi176.stdout:==> Vault server configuration: 2024-01-16T04:36:58.954 INFO:tasks.vault.client.0.smithi176.stdout: 2024-01-16T04:36:58.954 INFO:tasks.vault.client.0.smithi176.stdout: Api Address: http://0.0.0.0:8200 2024-01-16T04:36:58.954 INFO:tasks.vault.client.0.smithi176.stdout: Cgo: disabled 2024-01-16T04:36:58.954 INFO:tasks.vault.client.0.smithi176.stdout: Cluster Address: https://0.0.0.0:8201 2024-01-16T04:36:58.954 INFO:tasks.vault.client.0.smithi176.stdout: Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled") 2024-01-16T04:36:58.954 INFO:tasks.vault.client.0.smithi176.stdout: Log Level: info 2024-01-16T04:36:58.954 INFO:tasks.vault.client.0.smithi176.stdout: Mlock: supported: true, enabled: false 2024-01-16T04:36:58.954 INFO:tasks.vault.client.0.smithi176.stdout: Storage: inmem 2024-01-16T04:36:58.954 INFO:tasks.vault.client.0.smithi176.stdout: Version: Vault v1.2.2 2024-01-16T04:36:58.955 INFO:tasks.vault.client.0.smithi176.stdout: 2024-01-16T04:36:58.972 INFO:tasks.vault.client.0.smithi176.stdout:WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory 2024-01-16T04:36:58.972 INFO:tasks.vault.client.0.smithi176.stdout:and starts unsealed with a single unseal key. The root token is already 2024-01-16T04:36:58.972 INFO:tasks.vault.client.0.smithi176.stdout:authenticated to the CLI, so you can immediately begin using Vault. 2024-01-16T04:36:58.972 INFO:tasks.vault.client.0.smithi176.stdout: 2024-01-16T04:36:58.972 INFO:tasks.vault.client.0.smithi176.stdout:You may need to set the following environment variable: 2024-01-16T04:36:58.972 INFO:tasks.vault.client.0.smithi176.stdout: 2024-01-16T04:36:58.972 INFO:tasks.vault.client.0.smithi176.stdout: $ export VAULT_ADDR='http://0.0.0.0:8200' 2024-01-16T04:36:58.973 INFO:tasks.vault.client.0.smithi176.stdout: 2024-01-16T04:36:58.973 INFO:tasks.vault.client.0.smithi176.stdout:The unseal key and root token are displayed below in case you want to 2024-01-16T04:36:58.973 INFO:tasks.vault.client.0.smithi176.stdout:seal/unseal the Vault or re-authenticate. 2024-01-16T04:36:58.973 INFO:tasks.vault.client.0.smithi176.stdout: 2024-01-16T04:36:58.973 INFO:tasks.vault.client.0.smithi176.stdout:Unseal Key: QIjcc737POOYyVG1N02Yh2pH+9k8hCz4HQ/zqXCl4GM= 2024-01-16T04:36:58.973 INFO:tasks.vault.client.0.smithi176.stdout:Root Token: test_root_token 2024-01-16T04:36:58.973 INFO:tasks.vault.client.0.smithi176.stdout: 2024-01-16T04:36:58.973 INFO:tasks.vault.client.0.smithi176.stdout:Development mode should NOT be used in production installations! 2024-01-16T04:36:58.973 INFO:tasks.vault.client.0.smithi176.stdout: 2024-01-16T04:36:58.973 INFO:tasks.vault.client.0.smithi176.stdout:==> Vault server started! Log data will stream in below: 2024-01-16T04:36:58.973 INFO:tasks.vault.client.0.smithi176.stdout: 2024-01-16T04:36:58.974 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.954Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set 2024-01-16T04:36:58.974 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.955Z [INFO] core: security barrier not initialized 2024-01-16T04:36:58.974 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.955Z [INFO] core: security barrier initialized: shares=1 threshold=1 2024-01-16T04:36:58.974 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.955Z [INFO] core: post-unseal setup starting 2024-01-16T04:36:58.974 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.963Z [INFO] core: loaded wrapping token key 2024-01-16T04:36:58.974 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.963Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2024-01-16T04:36:58.974 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.963Z [INFO] core: no mounts; adding default mount table 2024-01-16T04:36:58.974 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.964Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2024-01-16T04:36:58.974 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.964Z [INFO] core: successfully mounted backend: type=system path=sys/ 2024-01-16T04:36:58.974 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.964Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.965Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.965Z [INFO] core: restoring leases 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.965Z [INFO] rollback: starting rollback manager 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.965Z [INFO] expiration: lease restore complete 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.966Z [INFO] identity: entities restored 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.966Z [INFO] identity: groups restored 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.966Z [INFO] core: post-unseal setup complete 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.966Z [INFO] core: root token generated 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.966Z [INFO] core: pre-seal teardown starting 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.966Z [INFO] rollback: stopping rollback manager 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.966Z [INFO] core: pre-seal teardown complete 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.967Z [INFO] core.cluster-listener: starting listener: listener_address=0.0.0.0:8201 2024-01-16T04:36:58.975 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.967Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201 2024-01-16T04:36:58.976 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.967Z [INFO] core: post-unseal setup starting 2024-01-16T04:36:58.976 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.967Z [INFO] core: loaded wrapping token key 2024-01-16T04:36:58.976 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.967Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2024-01-16T04:36:58.976 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.967Z [INFO] core: successfully mounted backend: type=system path=sys/ 2024-01-16T04:36:58.976 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.967Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2024-01-16T04:36:58.976 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.967Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2024-01-16T04:36:58.976 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.968Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2024-01-16T04:36:58.976 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.968Z [INFO] core: restoring leases 2024-01-16T04:36:58.977 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.968Z [INFO] rollback: starting rollback manager 2024-01-16T04:36:58.977 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.968Z [INFO] identity: entities restored 2024-01-16T04:36:58.977 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.968Z [INFO] expiration: lease restore complete 2024-01-16T04:36:58.977 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.968Z [INFO] identity: groups restored 2024-01-16T04:36:58.977 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.968Z [INFO] core: post-unseal setup complete 2024-01-16T04:36:58.977 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.968Z [INFO] core: vault is unsealed 2024-01-16T04:36:58.977 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.969Z [INFO] expiration: revoked lease: lease_id=auth/token/root/hca851d062852f19a3320d178df630816b599975c9121446d77203b3be483f1aa 2024-01-16T04:36:58.977 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.970Z [INFO] core: successful mount: namespace= path=secret/ type=kv 2024-01-16T04:36:58.978 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.980Z [INFO] secrets.kv.kv_20a83080: collecting keys to upgrade 2024-01-16T04:36:58.978 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.980Z [INFO] secrets.kv.kv_20a83080: done collecting keys: num_keys=1 2024-01-16T04:36:58.978 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:36:58.980Z [INFO] secrets.kv.kv_20a83080: upgrading keys finished 2024-01-16T04:37:08.906 INFO:tasks.vault:Mounting kv version 2 secrets engine 2024-01-16T04:37:08.906 INFO:tasks.vault:Send request to Vault: smithi176.front.sepia.ceph.com:8200 at /v1/sys/mounts/kv with token: test_root_token 2024-01-16T04:37:08.913 INFO:tasks.vault:b'' 2024-01-16T04:37:08.914 INFO:tasks.vault:Send request to Vault: smithi176.front.sepia.ceph.com:8200 at /v1/kv/data/my-key-1 with token: test_root_token 2024-01-16T04:37:08.914 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:37:08.917Z [INFO] core: successful mount: namespace= path=kv/ type=kv 2024-01-16T04:37:08.923 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:37:08.927Z [INFO] secrets.kv.kv_ab125866: collecting keys to upgrade 2024-01-16T04:37:08.923 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:37:08.928Z [INFO] secrets.kv.kv_ab125866: done collecting keys: num_keys=3 2024-01-16T04:37:08.923 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:37:08.928Z [INFO] secrets.kv.kv_ab125866: upgrading keys finished 2024-01-16T04:37:08.932 INFO:tasks.vault:b'{"request_id":"5ae344d0-8c82-1bed-90bd-42e01fe41bd5","lease_id":"","renewable":false,"lease_duration":0,"data":{"created_time":"2024-01-16T04:37:08.936540719Z","deletion_time":"","destroyed":false,"version":1},"wrap_info":null,"warnings":null,"auth":null}\n' 2024-01-16T04:37:08.932 INFO:tasks.vault:Send request to Vault: smithi176.front.sepia.ceph.com:8200 at /v1/kv/data/my-key-2 with token: test_root_token 2024-01-16T04:37:08.935 INFO:tasks.vault:b'{"request_id":"0fea816b-cc13-62a6-f864-f8a98f6a71a2","lease_id":"","renewable":false,"lease_duration":0,"data":{"created_time":"2024-01-16T04:37:08.940088979Z","deletion_time":"","destroyed":false,"version":1},"wrap_info":null,"warnings":null,"auth":null}\n' 2024-01-16T04:37:08.935 INFO:tasks.vault:secrets created 2024-01-16T04:37:08.936 INFO:teuthology.run_tasks:Running task rgw... 2024-01-16T04:37:08.949 DEBUG:tasks.rgw:config is {'client.0': {'use-vault-role': 'client.0'}} 2024-01-16T04:37:08.949 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2024-01-16T04:37:08.949 INFO:tasks.rgw:Creating data pools 2024-01-16T04:37:08.949 DEBUG:tasks.rgw:Obtaining remote for client client.0 2024-01-16T04:37:08.949 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2024-01-16T04:37:09.425 INFO:teuthology.orchestra.run.smithi176.stderr:pool 'default.rgw.buckets.data' created 2024-01-16T04:37:09.440 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2024-01-16T04:37:10.436 INFO:teuthology.orchestra.run.smithi176.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-01-16T04:37:10.451 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2024-01-16T04:37:11.436 INFO:teuthology.orchestra.run.smithi176.stderr:pool 'default.rgw.buckets.index' created 2024-01-16T04:37:11.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2024-01-16T04:37:12.442 INFO:teuthology.orchestra.run.smithi176.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-01-16T04:37:12.460 DEBUG:tasks.rgw:Pools created 2024-01-16T04:37:12.460 INFO:tasks.rgw:Starting rgw... 2024-01-16T04:37:12.460 INFO:tasks.rgw:rgw client.0 config is {'use-vault-role': 'client.0'} 2024-01-16T04:37:12.460 INFO:tasks.rgw:Using civetweb as radosgw frontend 2024-01-16T04:37:12.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo echo -n test_root_token | sudo tee /etc/ceph/vault-root-token 2024-01-16T04:37:12.489 INFO:teuthology.orchestra.run.smithi176.stdout:test_root_token 2024-01-16T04:37:12.489 INFO:tasks.rgw:Token file content 2024-01-16T04:37:12.489 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/ceph/vault-root-token 2024-01-16T04:37:12.547 INFO:teuthology.orchestra.run.smithi176.stdout:test_root_token 2024-01-16T04:37:12.547 INFO:tasks.rgw:Restrict access to token file 2024-01-16T04:37:12.547 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 600 /etc/ceph/vault-root-token 2024-01-16T04:37:12.614 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown ceph /etc/ceph/vault-root-token 2024-01-16T04:37:12.679 INFO:tasks.rgw.client.0:Restarting daemon 2024-01-16T04:37:12.680 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi176.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2024-01-16T04:37:12.722 INFO:tasks.rgw.client.0:Started 2024-01-16T04:37:12.722 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi176.front.sepia.ceph.com:80/ 2024-01-16T04:37:12.722 DEBUG:teuthology.orchestra.run.smithi176:> curl http://smithi176.front.sepia.ceph.com:80/ 2024-01-16T04:37:12.742 INFO:teuthology.orchestra.run.smithi176.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:37:12.743 INFO:teuthology.orchestra.run.smithi176.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:37:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-01-16T04:37:12.744 INFO:teuthology.orchestra.run.smithi176.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi176.front.sepia.ceph.com port 80: Connection refused 2024-01-16T04:37:13.745 DEBUG:teuthology.orchestra.run.smithi176:> curl http://smithi176.front.sepia.ceph.com:80/ 2024-01-16T04:37:13.769 INFO:teuthology.orchestra.run.smithi176.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:37:13.769 INFO:teuthology.orchestra.run.smithi176.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:37:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-01-16T04:37:13.771 INFO:teuthology.orchestra.run.smithi176.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi176.front.sepia.ceph.com port 80: Connection refused 2024-01-16T04:37:15.771 DEBUG:teuthology.orchestra.run.smithi176:> curl http://smithi176.front.sepia.ceph.com:80/ 2024-01-16T04:37:15.797 INFO:teuthology.orchestra.run.smithi176.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:37:15.797 INFO:teuthology.orchestra.run.smithi176.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:37:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-01-16T04:37:15.798 INFO:teuthology.orchestra.run.smithi176.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi176.front.sepia.ceph.com port 80: Connection refused 2024-01-16T04:37:19.799 DEBUG:teuthology.orchestra.run.smithi176:> curl http://smithi176.front.sepia.ceph.com:80/ 2024-01-16T04:37:19.824 INFO:teuthology.orchestra.run.smithi176.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:37:19.825 INFO:teuthology.orchestra.run.smithi176.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:37:19.826 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-01-16T04:37:19.826 INFO:teuthology.orchestra.run.smithi176.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi176.front.sepia.ceph.com port 80: Connection refused 2024-01-16T04:37:21.533 INFO:tasks.rgw.client.0.smithi176.stdout:2024-01-16T04:37:21.537+0000 7f3c4feaa880 -1 IMPORTANT: the civetweb frontend is now deprecated and will be removed in a future release 2024-01-16T04:37:27.827 DEBUG:teuthology.orchestra.run.smithi176:> curl http://smithi176.front.sepia.ceph.com:80/ 2024-01-16T04:37:27.854 INFO:teuthology.orchestra.run.smithi176.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:37:27.854 INFO:teuthology.orchestra.run.smithi176.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:37:27.854 INFO:teuthology.orchestra.run.smithi176.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-01-16T04:37:27.855 INFO:teuthology.orchestra.run.smithi176.stdout:anonymous 2024-01-16T04:37:27.855 INFO:teuthology.run_tasks:Running task s3tests... 2024-01-16T04:37:27.869 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-pacific', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}} 2024-01-16T04:37:27.869 INFO:tasks.s3tests:Downloading s3-tests... 2024-01-16T04:37:27.869 INFO:tasks.s3tests:Using branch 'ceph-pacific' for s3tests 2024-01-16T04:37:27.870 DEBUG:teuthology.orchestra.run.smithi176:> git clone -b ceph-pacific https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests 2024-01-16T04:37:27.917 INFO:teuthology.orchestra.run.smithi176.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests'... 2024-01-16T04:37:29.116 INFO:tasks.s3tests:Creating rgw users... 2024-01-16T04:37:29.117 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2024-01-16T04:37:29.117 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key KRDTGHFZGBRBRVNHGFXA --secret lGTWW2i4xX5oVOl/LvRpB41unUqQT4QAMXG384q1uPcXW/TPCHrF5g== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2024-01-16T04:37:29.211 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:37:29.212 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:37:29.218 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.222+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.223+0000 7f56f00a07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.223+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.223+0000 7f56f00a07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.223+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.223+0000 7f56f00a07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.223+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.223+0000 7f56f00a07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:29.228 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.231+0000 7f56f00a07c0 20 RGWRados::pool_iterate: got zonegroup_info.ddb5a393-8a31-4788-b116-e494d90f23e0 2024-01-16T04:37:29.228 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.231+0000 7f56f00a07c0 20 RGWRados::pool_iterate: got zone_info.9cd49eee-6ce1-47e3-8681-41594a956139 2024-01-16T04:37:29.228 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.231+0000 7f56f00a07c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:37:29.228 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.231+0000 7f56f00a07c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:37:29.228 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.231+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.229 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.232+0000 7f56f00a07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:29.229 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.232+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.229 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.232+0000 7f56f00a07c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:37:29.229 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.232+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.229 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.232+0000 7f56f00a07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:29.229 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.232+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.229 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.232+0000 7f56f00a07c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:37:29.229 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.232+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.233+0000 7f56f00a07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.233+0000 7f56f00a07c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:37:29.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.233+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.233+0000 7f56f00a07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.233+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.233+0000 7f56f00a07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.233+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.233+0000 7f56f00a07c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.233+0000 7f56f00a07c0 20 zonegroup default 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.233+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.234+0000 7f56f00a07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.234+0000 7f56f00a07c0 10 Cannot find current period zone using local zone 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.234+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.234+0000 7f56f00a07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.234+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.234+0000 7f56f00a07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:29.233 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.234+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.233 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.234+0000 7f56f00a07c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:37:29.233 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.234+0000 7f56f00a07c0 20 zone default found 2024-01-16T04:37:29.233 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.234+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.233 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.235+0000 7f56f00a07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.233 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.235+0000 7f56f00a07c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.233 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.235+0000 7f56f00a07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.233 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.235+0000 7f56f00a07c0 20 started sync module instance, tier type = 2024-01-16T04:37:29.233 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.235+0000 7f56f00a07c0 20 started zone id=9cd49eee-6ce1-47e3-8681-41594a956139 (name=default) with tier type = 2024-01-16T04:37:29.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.247+0000 7f56f00a07c0 20 add_watcher() i=0 2024-01-16T04:37:29.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.247+0000 7f56f00a07c0 20 add_watcher() i=1 2024-01-16T04:37:29.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.247+0000 7f56f00a07c0 20 add_watcher() i=2 2024-01-16T04:37:29.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.247+0000 7f56f00a07c0 20 add_watcher() i=3 2024-01-16T04:37:29.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.247+0000 7f56f00a07c0 20 add_watcher() i=4 2024-01-16T04:37:29.244 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.247+0000 7f56f00a07c0 20 add_watcher() i=5 2024-01-16T04:37:29.244 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.247+0000 7f56f00a07c0 20 add_watcher() i=6 2024-01-16T04:37:29.244 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.248+0000 7f56f00a07c0 20 add_watcher() i=7 2024-01-16T04:37:29.244 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.248+0000 7f56f00a07c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:37:29.246 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.249+0000 7f5627fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:37:29.246 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.249+0000 7f56f00a07c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:37:29.246 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.249+0000 7f56f00a07c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:37:29.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.253+0000 7f56f00a07c0 5 note: GC not initialized 2024-01-16T04:37:29.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.254+0000 7f561effd700 20 reqs_thread_entry: start 2024-01-16T04:37:29.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.254+0000 7f561cff9700 20 reqs_thread_entry: start 2024-01-16T04:37:29.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.254+0000 7f56f00a07c0 20 init_complete bucket index max shards: 11 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f56f00a07c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f560affd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:37:59 2024 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f560a7fc700 20 reqs_thread_entry: start 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f56f00a07c0 20 get_system_obj_state: rctx=0x7ffcc812b568 obj=default.rgw.meta:users.uid:foo.client.0 state=0x561319a9cd50 s->prefetch_data=0 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f56f00a07c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f56f00a07c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f56f00a07c0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f56f00a07c0 20 get_system_obj_state: rctx=0x7ffcc812b578 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x561319a9cd50 s->prefetch_data=0 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f56f00a07c0 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f56f00a07c0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f56f00a07c0 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f56f00a07c0 20 get_system_obj_state: rctx=0x7ffcc812b578 obj=default.rgw.meta:users.keys:KRDTGHFZGBRBRVNHGFXA state=0x561319a9cd50 s->prefetch_data=0 2024-01-16T04:37:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.255+0000 7f56f00a07c0 10 cache get: name=default.rgw.meta+users.keys+KRDTGHFZGBRBRVNHGFXA : miss 2024-01-16T04:37:29.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.256+0000 7f56f00a07c0 10 cache put: name=default.rgw.meta+users.keys+KRDTGHFZGBRBRVNHGFXA info.flags=0x0 2024-01-16T04:37:29.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.256+0000 7f56f00a07c0 10 adding default.rgw.meta+users.keys+KRDTGHFZGBRBRVNHGFXA to cache LRU end 2024-01-16T04:37:29.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.256+0000 7f56f00a07c0 20 get_system_obj_state: rctx=0x7ffcc812ae98 obj=default.rgw.meta:users.keys:KRDTGHFZGBRBRVNHGFXA state=0x561319a9cd50 s->prefetch_data=0 2024-01-16T04:37:29.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.256+0000 7f56f00a07c0 10 cache get: name=default.rgw.meta+users.keys+KRDTGHFZGBRBRVNHGFXA : hit (negative entry) 2024-01-16T04:37:29.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.256+0000 7f56f00a07c0 20 get_system_obj_state: rctx=0x7ffcc812b1d8 obj=default.rgw.meta:users.keys:KRDTGHFZGBRBRVNHGFXA state=0x561319a9cd50 s->prefetch_data=0 2024-01-16T04:37:29.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.256+0000 7f56f00a07c0 10 cache get: name=default.rgw.meta+users.keys+KRDTGHFZGBRBRVNHGFXA : hit (negative entry) 2024-01-16T04:37:29.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.257+0000 7f56f00a07c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-01-16T04:37:29.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.257+0000 7f56f00a07c0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:37:29.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.257+0000 7f56f00a07c0 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-01-16T04:37:29.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.257+0000 7f56d91fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149696 cookie 94640036242768 notifier 4202 bl.length()=640 2024-01-16T04:37:29.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.257+0000 7f56d91fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-01-16T04:37:29.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.257+0000 7f56d91fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:37:29.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.259+0000 7f56f00a07c0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2024-01-16T04:37:29.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.259+0000 7f56f00a07c0 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2024-01-16T04:37:29.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.259+0000 7f56f00a07c0 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2024-01-16T04:37:29.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.259+0000 7f56d91fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149696 cookie 94640036262864 notifier 4202 bl.length()=194 2024-01-16T04:37:29.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.259+0000 7f56d91fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2024-01-16T04:37:29.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.259+0000 7f56d91fc700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2024-01-16T04:37:29.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.260+0000 7f56f00a07c0 10 cache put: name=default.rgw.meta+users.keys+KRDTGHFZGBRBRVNHGFXA info.flags=0x7 2024-01-16T04:37:29.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.260+0000 7f56f00a07c0 10 moving default.rgw.meta+users.keys+KRDTGHFZGBRBRVNHGFXA to cache LRU end 2024-01-16T04:37:29.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.260+0000 7f56f00a07c0 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.keys:KRDTGHFZGBRBRVNHGFXA, ofs0, ns] 2024-01-16T04:37:29.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.261+0000 7f56d89fb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149697 cookie 94640036256736 notifier 4202 bl.length()=186 2024-01-16T04:37:29.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.261+0000 7f56d89fb700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+KRDTGHFZGBRBRVNHGFXA info.flags=0x7 2024-01-16T04:37:29.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.261+0000 7f56d89fb700 10 rgw watcher librados: moving default.rgw.meta+users.keys+KRDTGHFZGBRBRVNHGFXA to cache LRU end 2024-01-16T04:37:29.258 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-01-16T04:37:29.258 INFO:teuthology.orchestra.run.smithi176.stdout: "user_id": "foo.client.0", 2024-01-16T04:37:29.258 INFO:teuthology.orchestra.run.smithi176.stdout: "display_name": "Mr. foo.client.0", 2024-01-16T04:37:29.258 INFO:teuthology.orchestra.run.smithi176.stdout: "email": "foo.client.0+test@test.test", 2024-01-16T04:37:29.258 INFO:teuthology.orchestra.run.smithi176.stdout: "suspended": 0, 2024-01-16T04:37:29.258 INFO:teuthology.orchestra.run.smithi176.stdout: "max_buckets": 1000, 2024-01-16T04:37:29.258 INFO:teuthology.orchestra.run.smithi176.stdout: "subusers": [], 2024-01-16T04:37:29.258 INFO:teuthology.orchestra.run.smithi176.stdout: "keys": [ 2024-01-16T04:37:29.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-01-16T04:37:29.258 INFO:teuthology.orchestra.run.smithi176.stdout: "user": "foo.client.0", 2024-01-16T04:37:29.258 INFO:teuthology.orchestra.run.smithi176.stdout: "access_key": "KRDTGHFZGBRBRVNHGFXA", 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: "secret_key": "lGTWW2i4xX5oVOl/LvRpB41unUqQT4QAMXG384q1uPcXW/TPCHrF5g==" 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: "swift_keys": [], 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: "caps": [ 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "user-policy", 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: "perm": "*" 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: "op_mask": "read, write, delete", 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: "default_placement": "", 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: "default_storage_class": "", 2024-01-16T04:37:29.259 INFO:teuthology.orchestra.run.smithi176.stdout: "placement_tags": [], 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: "bucket_quota": { 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: "enabled": false, 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: "check_on_raw": false, 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: "max_size": -1, 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: "max_size_kb": 0, 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: "max_objects": -1 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: "user_quota": { 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: "enabled": false, 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: "check_on_raw": false, 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: "max_size": -1, 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: "max_size_kb": 0, 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: "max_objects": -1 2024-01-16T04:37:29.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-01-16T04:37:29.261 INFO:teuthology.orchestra.run.smithi176.stdout: "temp_url_keys": [], 2024-01-16T04:37:29.261 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "rgw", 2024-01-16T04:37:29.261 INFO:teuthology.orchestra.run.smithi176.stdout: "mfa_ids": [] 2024-01-16T04:37:29.261 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-01-16T04:37:29.261 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:37:29.261 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.263+0000 7f56f00a07c0 20 remove_watcher() i=0 2024-01-16T04:37:29.261 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.263+0000 7f56f00a07c0 2 removed watcher, disabling cache 2024-01-16T04:37:29.262 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.265+0000 7f56f00a07c0 20 remove_watcher() i=1 2024-01-16T04:37:29.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.266+0000 7f56f00a07c0 20 remove_watcher() i=2 2024-01-16T04:37:29.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.270+0000 7f56f00a07c0 20 remove_watcher() i=3 2024-01-16T04:37:29.269 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.272+0000 7f56f00a07c0 20 remove_watcher() i=4 2024-01-16T04:37:29.271 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.275+0000 7f56f00a07c0 20 remove_watcher() i=5 2024-01-16T04:37:29.272 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.276+0000 7f56f00a07c0 20 remove_watcher() i=6 2024-01-16T04:37:29.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.278+0000 7f56f00a07c0 20 remove_watcher() i=7 2024-01-16T04:37:29.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.278+0000 7f5627fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:37:29.281 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 0892554626 --totp-seed SM6RSQXJXGGBP3XXM4QLCHCPRISRLH242M7VSO52T5P4XHENFT2PFO44NY266PKQ --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-01-16T04:37:29.330 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:37:29.330 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:37:29.335 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.339+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.336 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.340+0000 7f0ad36a57c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.336 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.340+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.336 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.340+0000 7f0ad36a57c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.336 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.340+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.337 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.340+0000 7f0ad36a57c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.337 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.340+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.337 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.340+0000 7f0ad36a57c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:29.345 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.347+0000 7f0ad36a57c0 20 RGWRados::pool_iterate: got zonegroup_info.ddb5a393-8a31-4788-b116-e494d90f23e0 2024-01-16T04:37:29.345 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.347+0000 7f0ad36a57c0 20 RGWRados::pool_iterate: got zone_info.9cd49eee-6ce1-47e3-8681-41594a956139 2024-01-16T04:37:29.345 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.347+0000 7f0ad36a57c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:37:29.345 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.347+0000 7f0ad36a57c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:37:29.345 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.347+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.345 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.347+0000 7f0ad36a57c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:29.345 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.347+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.345 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.347+0000 7f0ad36a57c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:37:29.345 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.347+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.345 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.348+0000 7f0ad36a57c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:29.346 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.348+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.346 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.348+0000 7f0ad36a57c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:37:29.346 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.348+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.346 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.348+0000 7f0ad36a57c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.346 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.348+0000 7f0ad36a57c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:37:29.346 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.348+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.346 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.348+0000 7f0ad36a57c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.346 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.348+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.346 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.348+0000 7f0ad36a57c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:29.346 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.348+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.348 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.349+0000 7f0ad36a57c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:37:29.348 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.349+0000 7f0ad36a57c0 20 zonegroup default 2024-01-16T04:37:29.348 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.349+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.348 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.349+0000 7f0ad36a57c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.348 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.349+0000 7f0ad36a57c0 10 Cannot find current period zone using local zone 2024-01-16T04:37:29.348 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.349+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.348 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.349+0000 7f0ad36a57c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.348 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.349+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.348 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.349+0000 7f0ad36a57c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:29.349 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.349+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.349 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.350+0000 7f0ad36a57c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:37:29.349 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.350+0000 7f0ad36a57c0 20 zone default found 2024-01-16T04:37:29.349 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.350+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.349 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.350+0000 7f0ad36a57c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.349 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.350+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.349 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.350+0000 7f0ad36a57c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:29.349 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.350+0000 7f0ad36a57c0 20 started sync module instance, tier type = 2024-01-16T04:37:29.349 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.350+0000 7f0ad36a57c0 20 started zone id=9cd49eee-6ce1-47e3-8681-41594a956139 (name=default) with tier type = 2024-01-16T04:37:29.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.379+0000 7f0ad36a57c0 20 add_watcher() i=0 2024-01-16T04:37:29.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.379+0000 7f0ad36a57c0 20 add_watcher() i=1 2024-01-16T04:37:29.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.379+0000 7f0ad36a57c0 20 add_watcher() i=2 2024-01-16T04:37:29.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.379+0000 7f0ad36a57c0 20 add_watcher() i=3 2024-01-16T04:37:29.377 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.379+0000 7f0ad36a57c0 20 add_watcher() i=4 2024-01-16T04:37:29.377 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.379+0000 7f0ad36a57c0 20 add_watcher() i=5 2024-01-16T04:37:29.377 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.379+0000 7f0ad36a57c0 20 add_watcher() i=6 2024-01-16T04:37:29.378 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.382+0000 7f0ad36a57c0 20 add_watcher() i=7 2024-01-16T04:37:29.378 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.382+0000 7f0ad36a57c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:37:29.381 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.384+0000 7f0a8ffe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:37:29.381 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.385+0000 7f0ad36a57c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:37:29.381 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.385+0000 7f0ad36a57c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:37:29.385 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.388+0000 7f0ad36a57c0 5 note: GC not initialized 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.389+0000 7f09feffd700 20 reqs_thread_entry: start 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.389+0000 7f09fcff9700 20 reqs_thread_entry: start 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.389+0000 7f0ad36a57c0 20 init_complete bucket index max shards: 11 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.389+0000 7f0ad36a57c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.389+0000 7f09f2ffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:37:59 2024 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.390+0000 7f09f27fc700 20 reqs_thread_entry: start 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.390+0000 7f0ad36a57c0 20 get_system_obj_state: rctx=0x7ffddef28e18 obj=default.rgw.meta:users.uid:foo.client.0 state=0x5633f7fd7d50 s->prefetch_data=0 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.390+0000 7f0ad36a57c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.390+0000 7f0ad36a57c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.390+0000 7f0ad36a57c0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.390+0000 7f0ad36a57c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:37:29.387 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.390+0000 7f0ad36a57c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:37:29.388 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.390+0000 7f0ad36a57c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:37:29.388 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.390+0000 7f0ad36a57c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:29.388 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.391+0000 7f0ad36a57c0 20 rados_obj.operate() r=0 bl.length=455 2024-01-16T04:37:29.388 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.391+0000 7f0ad36a57c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x11 2024-01-16T04:37:29.388 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:29.391+0000 7f0ad36a57c0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:37:30.508 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.511+0000 7f0ad36a57c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-01-16T04:37:30.508 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.511+0000 7f0ad36a57c0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:37:30.508 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.511+0000 7f0ad36a57c0 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-01-16T04:37:30.508 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.512+0000 7f0aaffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084289 cookie 94781205222704 notifier 4208 bl.length()=654 2024-01-16T04:37:30.508 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.512+0000 7f0aaffff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-01-16T04:37:30.508 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.512+0000 7f0aaffff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:37:30.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.514+0000 7f0ad36a57c0 20 remove_watcher() i=0 2024-01-16T04:37:30.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.514+0000 7f0ad36a57c0 2 removed watcher, disabling cache 2024-01-16T04:37:30.512 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.515+0000 7f0ad36a57c0 20 remove_watcher() i=1 2024-01-16T04:37:30.513 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.516+0000 7f0ad36a57c0 20 remove_watcher() i=2 2024-01-16T04:37:30.514 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.517+0000 7f0ad36a57c0 20 remove_watcher() i=3 2024-01-16T04:37:30.515 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.519+0000 7f0ad36a57c0 20 remove_watcher() i=4 2024-01-16T04:37:30.516 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.520+0000 7f0ad36a57c0 20 remove_watcher() i=5 2024-01-16T04:37:30.517 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.521+0000 7f0ad36a57c0 20 remove_watcher() i=6 2024-01-16T04:37:30.518 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.522+0000 7f0ad36a57c0 20 remove_watcher() i=7 2024-01-16T04:37:30.519 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.522+0000 7f0a8ffe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:37:30.524 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2024-01-16T04:37:30.524 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key KIFEZSRMUIJENLUODLLJ --secret KsDuEzLJAHEnKZHySCySiqmOKX43VZAhdZwkXcMpMW6ypUZb54D3BA== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2024-01-16T04:37:30.576 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:37:30.576 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:37:30.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.585+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.583 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.586+0000 7f7ef39537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.584 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.586+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.584 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.587+0000 7f7ef39537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.584 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.587+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.584 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.587+0000 7f7ef39537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.584 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.587+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.584 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.587+0000 7f7ef39537c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.595+0000 7f7ef39537c0 20 RGWRados::pool_iterate: got zonegroup_info.ddb5a393-8a31-4788-b116-e494d90f23e0 2024-01-16T04:37:30.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.595+0000 7f7ef39537c0 20 RGWRados::pool_iterate: got zone_info.9cd49eee-6ce1-47e3-8681-41594a956139 2024-01-16T04:37:30.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.595+0000 7f7ef39537c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:37:30.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.595+0000 7f7ef39537c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:37:30.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.595+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.595+0000 7f7ef39537c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.595+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.595+0000 7f7ef39537c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:37:30.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.595+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.595+0000 7f7ef39537c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.595+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.596+0000 7f7ef39537c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.596+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.596+0000 7f7ef39537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.596+0000 7f7ef39537c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.596+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.596+0000 7f7ef39537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.596+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.596+0000 7f7ef39537c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.596+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.596+0000 7f7ef39537c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.596+0000 7f7ef39537c0 20 zonegroup default 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.596+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.597+0000 7f7ef39537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.597+0000 7f7ef39537c0 10 Cannot find current period zone using local zone 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.597+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.597+0000 7f7ef39537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.597+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.597+0000 7f7ef39537c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.597+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.597+0000 7f7ef39537c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.597+0000 7f7ef39537c0 20 zone default found 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.597+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.598+0000 7f7ef39537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.598+0000 7f7ef39537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.598+0000 7f7ef39537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.598+0000 7f7ef39537c0 20 started sync module instance, tier type = 2024-01-16T04:37:30.596 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.598+0000 7f7ef39537c0 20 started zone id=9cd49eee-6ce1-47e3-8681-41594a956139 (name=default) with tier type = 2024-01-16T04:37:30.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.611+0000 7f7ef39537c0 20 add_watcher() i=0 2024-01-16T04:37:30.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.611+0000 7f7ef39537c0 20 add_watcher() i=1 2024-01-16T04:37:30.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.611+0000 7f7ef39537c0 20 add_watcher() i=2 2024-01-16T04:37:30.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.611+0000 7f7ef39537c0 20 add_watcher() i=3 2024-01-16T04:37:30.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.611+0000 7f7ef39537c0 20 add_watcher() i=4 2024-01-16T04:37:30.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.611+0000 7f7ef39537c0 20 add_watcher() i=5 2024-01-16T04:37:30.609 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.611+0000 7f7ef39537c0 20 add_watcher() i=6 2024-01-16T04:37:30.610 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.614+0000 7f7ef39537c0 20 add_watcher() i=7 2024-01-16T04:37:30.610 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.614+0000 7f7ef39537c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:37:30.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.615+0000 7f7e2bfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:37:30.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.615+0000 7f7ef39537c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:37:30.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.615+0000 7f7ef39537c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:37:30.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.619+0000 7f7ef39537c0 5 note: GC not initialized 2024-01-16T04:37:30.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.620+0000 7f7e1affd700 20 reqs_thread_entry: start 2024-01-16T04:37:30.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.620+0000 7f7e197fa700 20 reqs_thread_entry: start 2024-01-16T04:37:30.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.620+0000 7f7ef39537c0 20 init_complete bucket index max shards: 11 2024-01-16T04:37:30.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7ef39537c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:37:30.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7e12ffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:38:01 2024 2024-01-16T04:37:30.618 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7e127fc700 20 reqs_thread_entry: start 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7ef39537c0 20 get_system_obj_state: rctx=0x7ffe6fbb0fb8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x562dd78d1d50 s->prefetch_data=0 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7ef39537c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7ef39537c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7ef39537c0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7ef39537c0 20 get_system_obj_state: rctx=0x7ffe6fbb0fc8 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x562dd78d1d50 s->prefetch_data=0 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7ef39537c0 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7ef39537c0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7ef39537c0 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7ef39537c0 20 get_system_obj_state: rctx=0x7ffe6fbb0fc8 obj=default.rgw.meta:users.keys:KIFEZSRMUIJENLUODLLJ state=0x562dd78d1d50 s->prefetch_data=0 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.621+0000 7f7ef39537c0 10 cache get: name=default.rgw.meta+users.keys+KIFEZSRMUIJENLUODLLJ : miss 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.622+0000 7f7ef39537c0 10 cache put: name=default.rgw.meta+users.keys+KIFEZSRMUIJENLUODLLJ info.flags=0x0 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.622+0000 7f7ef39537c0 10 adding default.rgw.meta+users.keys+KIFEZSRMUIJENLUODLLJ to cache LRU end 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.622+0000 7f7ef39537c0 20 get_system_obj_state: rctx=0x7ffe6fbb08e8 obj=default.rgw.meta:users.keys:KIFEZSRMUIJENLUODLLJ state=0x562dd78d1d50 s->prefetch_data=0 2024-01-16T04:37:30.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.622+0000 7f7ef39537c0 10 cache get: name=default.rgw.meta+users.keys+KIFEZSRMUIJENLUODLLJ : hit (negative entry) 2024-01-16T04:37:30.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.622+0000 7f7ef39537c0 20 get_system_obj_state: rctx=0x7ffe6fbb0c28 obj=default.rgw.meta:users.keys:KIFEZSRMUIJENLUODLLJ state=0x562dd78d1d50 s->prefetch_data=0 2024-01-16T04:37:30.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.622+0000 7f7ef39537c0 10 cache get: name=default.rgw.meta+users.keys+KIFEZSRMUIJENLUODLLJ : hit (negative entry) 2024-01-16T04:37:30.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.623+0000 7f7ef39537c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-01-16T04:37:30.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.623+0000 7f7ef39537c0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:37:30.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.623+0000 7f7ef39537c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-01-16T04:37:30.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.623+0000 7f7edcaaf700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084290 cookie 94754891188688 notifier 4214 bl.length()=640 2024-01-16T04:37:30.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.623+0000 7f7edcaaf700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-01-16T04:37:30.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.623+0000 7f7edcaaf700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:37:30.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.625+0000 7f7ef39537c0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2024-01-16T04:37:30.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.625+0000 7f7ef39537c0 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2024-01-16T04:37:30.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.625+0000 7f7ef39537c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2024-01-16T04:37:30.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.625+0000 7f7edcaaf700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084291 cookie 94754891188688 notifier 4214 bl.length()=194 2024-01-16T04:37:30.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.625+0000 7f7edcaaf700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2024-01-16T04:37:30.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.625+0000 7f7edcaaf700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2024-01-16T04:37:30.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.626+0000 7f7ef39537c0 10 cache put: name=default.rgw.meta+users.keys+KIFEZSRMUIJENLUODLLJ info.flags=0x7 2024-01-16T04:37:30.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.626+0000 7f7ef39537c0 10 moving default.rgw.meta+users.keys+KIFEZSRMUIJENLUODLLJ to cache LRU end 2024-01-16T04:37:30.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.626+0000 7f7ef39537c0 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.keys:KIFEZSRMUIJENLUODLLJ, ofs0, ns] 2024-01-16T04:37:30.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.626+0000 7f7ed7fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084292 cookie 94754891195408 notifier 4214 bl.length()=186 2024-01-16T04:37:30.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.626+0000 7f7ed7fff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+KIFEZSRMUIJENLUODLLJ info.flags=0x7 2024-01-16T04:37:30.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.626+0000 7f7ed7fff700 10 rgw watcher librados: moving default.rgw.meta+users.keys+KIFEZSRMUIJENLUODLLJ to cache LRU end 2024-01-16T04:37:30.624 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-01-16T04:37:30.624 INFO:teuthology.orchestra.run.smithi176.stdout: "user_id": "bar.client.0", 2024-01-16T04:37:30.624 INFO:teuthology.orchestra.run.smithi176.stdout: "display_name": "Mr. bar.client.0", 2024-01-16T04:37:30.624 INFO:teuthology.orchestra.run.smithi176.stdout: "email": "bar.client.0+test@test.test", 2024-01-16T04:37:30.624 INFO:teuthology.orchestra.run.smithi176.stdout: "suspended": 0, 2024-01-16T04:37:30.624 INFO:teuthology.orchestra.run.smithi176.stdout: "max_buckets": 1000, 2024-01-16T04:37:30.624 INFO:teuthology.orchestra.run.smithi176.stdout: "subusers": [], 2024-01-16T04:37:30.624 INFO:teuthology.orchestra.run.smithi176.stdout: "keys": [ 2024-01-16T04:37:30.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-01-16T04:37:30.624 INFO:teuthology.orchestra.run.smithi176.stdout: "user": "bar.client.0", 2024-01-16T04:37:30.624 INFO:teuthology.orchestra.run.smithi176.stdout: "access_key": "KIFEZSRMUIJENLUODLLJ", 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: "secret_key": "KsDuEzLJAHEnKZHySCySiqmOKX43VZAhdZwkXcMpMW6ypUZb54D3BA==" 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: "swift_keys": [], 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: "caps": [ 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "user-policy", 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: "perm": "*" 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: "op_mask": "read, write, delete", 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: "default_placement": "", 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: "default_storage_class": "", 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: "placement_tags": [], 2024-01-16T04:37:30.625 INFO:teuthology.orchestra.run.smithi176.stdout: "bucket_quota": { 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "enabled": false, 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "check_on_raw": false, 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "max_size": -1, 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "max_size_kb": 0, 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "max_objects": -1 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "user_quota": { 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "enabled": false, 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "check_on_raw": false, 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "max_size": -1, 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "max_size_kb": 0, 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "max_objects": -1 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "temp_url_keys": [], 2024-01-16T04:37:30.626 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "rgw", 2024-01-16T04:37:30.627 INFO:teuthology.orchestra.run.smithi176.stdout: "mfa_ids": [] 2024-01-16T04:37:30.627 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-01-16T04:37:30.627 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:37:30.627 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.629+0000 7f7ef39537c0 20 remove_watcher() i=0 2024-01-16T04:37:30.627 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.629+0000 7f7ef39537c0 2 removed watcher, disabling cache 2024-01-16T04:37:30.627 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.629+0000 7f7ef39537c0 20 remove_watcher() i=1 2024-01-16T04:37:30.628 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.631+0000 7f7ef39537c0 20 remove_watcher() i=2 2024-01-16T04:37:30.630 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.634+0000 7f7ef39537c0 20 remove_watcher() i=3 2024-01-16T04:37:30.633 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.637+0000 7f7ef39537c0 20 remove_watcher() i=4 2024-01-16T04:37:30.634 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.638+0000 7f7ef39537c0 20 remove_watcher() i=5 2024-01-16T04:37:30.636 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.639+0000 7f7ef39537c0 20 remove_watcher() i=6 2024-01-16T04:37:30.639 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.642+0000 7f7ef39537c0 20 remove_watcher() i=7 2024-01-16T04:37:30.639 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.643+0000 7f7e2bfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:37:30.645 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 9660752566 --totp-seed WQWVE3PLYH2TQ5D6FLUCFLUGBJMMBW5YAONFGW5FPH4ZSC3VJ5NBRYCY46VUJGTO --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-01-16T04:37:30.693 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:37:30.693 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:37:30.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.703+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.703+0000 7fbfef6537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.703+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.704+0000 7fbfef6537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.704+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.704+0000 7fbfef6537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.704+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.704+0000 7fbfef6537c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.711+0000 7fbfef6537c0 20 RGWRados::pool_iterate: got zonegroup_info.ddb5a393-8a31-4788-b116-e494d90f23e0 2024-01-16T04:37:30.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.711+0000 7fbfef6537c0 20 RGWRados::pool_iterate: got zone_info.9cd49eee-6ce1-47e3-8681-41594a956139 2024-01-16T04:37:30.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.711+0000 7fbfef6537c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:37:30.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.711+0000 7fbfef6537c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:37:30.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.711+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.711+0000 7fbfef6537c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.711+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.712+0000 7fbfef6537c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:37:30.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.712+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.712+0000 7fbfef6537c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.712+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.712+0000 7fbfef6537c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:37:30.710 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.712+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.710 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.712+0000 7fbfef6537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.710 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.712+0000 7fbfef6537c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:37:30.710 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.712+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.710 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.712+0000 7fbfef6537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.710 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.712+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.713+0000 7fbfef6537c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.713+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.713+0000 7fbfef6537c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.713+0000 7fbfef6537c0 20 zonegroup default 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.713+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.713+0000 7fbfef6537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.713+0000 7fbfef6537c0 10 Cannot find current period zone using local zone 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.713+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.713+0000 7fbfef6537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.713+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.714+0000 7fbfef6537c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.714+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.714+0000 7fbfef6537c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:37:30.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.714+0000 7fbfef6537c0 20 zone default found 2024-01-16T04:37:30.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.714+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.714+0000 7fbfef6537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.714+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.714+0000 7fbfef6537c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.714+0000 7fbfef6537c0 20 started sync module instance, tier type = 2024-01-16T04:37:30.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.714+0000 7fbfef6537c0 20 started zone id=9cd49eee-6ce1-47e3-8681-41594a956139 (name=default) with tier type = 2024-01-16T04:37:30.734 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.737+0000 7fbfef6537c0 20 add_watcher() i=0 2024-01-16T04:37:30.734 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.737+0000 7fbfef6537c0 20 add_watcher() i=1 2024-01-16T04:37:30.734 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.737+0000 7fbfef6537c0 20 add_watcher() i=2 2024-01-16T04:37:30.734 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.737+0000 7fbfef6537c0 20 add_watcher() i=3 2024-01-16T04:37:30.734 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.737+0000 7fbfef6537c0 20 add_watcher() i=4 2024-01-16T04:37:30.734 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.737+0000 7fbfef6537c0 20 add_watcher() i=5 2024-01-16T04:37:30.734 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.737+0000 7fbfef6537c0 20 add_watcher() i=6 2024-01-16T04:37:30.736 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.740+0000 7fbfef6537c0 20 add_watcher() i=7 2024-01-16T04:37:30.736 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.740+0000 7fbfef6537c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:37:30.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.741+0000 7fbfabfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:37:30.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.742+0000 7fbfef6537c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:37:30.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.742+0000 7fbfef6537c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:37:30.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.746+0000 7fbfef6537c0 5 note: GC not initialized 2024-01-16T04:37:30.745 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbf1affd700 20 reqs_thread_entry: start 2024-01-16T04:37:30.745 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbf18ff9700 20 reqs_thread_entry: start 2024-01-16T04:37:30.745 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbfef6537c0 20 init_complete bucket index max shards: 11 2024-01-16T04:37:30.745 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbfef6537c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:37:30.745 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbf0effd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:38:00 2024 2024-01-16T04:37:30.745 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:37:30.745 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbf0e7fc700 20 reqs_thread_entry: start 2024-01-16T04:37:30.745 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbfef6537c0 20 get_system_obj_state: rctx=0x7ffdc0af4b98 obj=default.rgw.meta:users.uid:bar.client.0 state=0x5599b37d9d50 s->prefetch_data=0 2024-01-16T04:37:30.745 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbfef6537c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-01-16T04:37:30.745 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbfef6537c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2024-01-16T04:37:30.746 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbfef6537c0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:37:30.746 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbfef6537c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:37:30.746 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbfef6537c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:37:30.746 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.747+0000 7fbfef6537c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:37:30.746 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.748+0000 7fbfef6537c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.746 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.748+0000 7fbfef6537c0 20 rados_obj.operate() r=0 bl.length=455 2024-01-16T04:37:30.746 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.748+0000 7fbfef6537c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x11 2024-01-16T04:37:30.746 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.748+0000 7fbfef6537c0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:37:30.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.750+0000 7fbfef6537c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-01-16T04:37:30.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.750+0000 7fbfef6537c0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:37:30.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.750+0000 7fbfef6537c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-01-16T04:37:30.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.750+0000 7fbfcbfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084293 cookie 94118631028384 notifier 4220 bl.length()=654 2024-01-16T04:37:30.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.750+0000 7fbfcbfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-01-16T04:37:30.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.750+0000 7fbfcbfff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:37:30.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.752+0000 7fbfef6537c0 20 remove_watcher() i=0 2024-01-16T04:37:30.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.752+0000 7fbfef6537c0 2 removed watcher, disabling cache 2024-01-16T04:37:30.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.753+0000 7fbfef6537c0 20 remove_watcher() i=1 2024-01-16T04:37:30.751 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.754+0000 7fbfef6537c0 20 remove_watcher() i=2 2024-01-16T04:37:30.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.757+0000 7fbfef6537c0 20 remove_watcher() i=3 2024-01-16T04:37:30.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.760+0000 7fbfef6537c0 20 remove_watcher() i=4 2024-01-16T04:37:30.759 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.762+0000 7fbfef6537c0 20 remove_watcher() i=5 2024-01-16T04:37:30.760 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.763+0000 7fbfef6537c0 20 remove_watcher() i=6 2024-01-16T04:37:30.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.764+0000 7fbfef6537c0 20 remove_watcher() i=7 2024-01-16T04:37:30.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.764+0000 7fbfabfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:37:30.766 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2024-01-16T04:37:30.766 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key PLSGDJEUQHITTOSCZDQL --secret 00xMWi+7SMUHJ790u01K/K8/ikNoNcTpp8tMYG5q8aEYTc2sntPwDw== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2024-01-16T04:37:30.815 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:37:30.815 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:37:30.821 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.824+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.825+0000 7faf4c1fc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.825+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.826+0000 7faf4c1fc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.826+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.826+0000 7faf4c1fc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.826+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.826+0000 7faf4c1fc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.834 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.833+0000 7faf4c1fc7c0 20 RGWRados::pool_iterate: got zonegroup_info.ddb5a393-8a31-4788-b116-e494d90f23e0 2024-01-16T04:37:30.834 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.833+0000 7faf4c1fc7c0 20 RGWRados::pool_iterate: got zone_info.9cd49eee-6ce1-47e3-8681-41594a956139 2024-01-16T04:37:30.834 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.833+0000 7faf4c1fc7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:37:30.834 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.833+0000 7faf4c1fc7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:37:30.835 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.833+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.835 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.833+0000 7faf4c1fc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.835 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.833+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.835 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.833+0000 7faf4c1fc7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:37:30.835 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.833+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.835 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.834+0000 7faf4c1fc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.835 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.834+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.836 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.834+0000 7faf4c1fc7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:37:30.836 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.834+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.836 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.834+0000 7faf4c1fc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.836 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.834+0000 7faf4c1fc7c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:37:30.836 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.834+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.836 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.834+0000 7faf4c1fc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.837 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.834+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.837 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.834+0000 7faf4c1fc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.837 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.834+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.837 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.835+0000 7faf4c1fc7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:37:30.837 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.835+0000 7faf4c1fc7c0 20 zonegroup default 2024-01-16T04:37:30.837 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.835+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.837 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.835+0000 7faf4c1fc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.838 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.835+0000 7faf4c1fc7c0 10 Cannot find current period zone using local zone 2024-01-16T04:37:30.838 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.835+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.838 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.835+0000 7faf4c1fc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.838 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.835+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.838 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.835+0000 7faf4c1fc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.838 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.835+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.839 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.836+0000 7faf4c1fc7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:37:30.839 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.836+0000 7faf4c1fc7c0 20 zone default found 2024-01-16T04:37:30.839 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.836+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.839 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.836+0000 7faf4c1fc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.839 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.836+0000 7faf4c1fc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.839 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.836+0000 7faf4c1fc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.840 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.836+0000 7faf4c1fc7c0 20 started sync module instance, tier type = 2024-01-16T04:37:30.840 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.836+0000 7faf4c1fc7c0 20 started zone id=9cd49eee-6ce1-47e3-8681-41594a956139 (name=default) with tier type = 2024-01-16T04:37:30.844 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.848+0000 7faf4c1fc7c0 20 add_watcher() i=0 2024-01-16T04:37:30.845 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.848+0000 7faf4c1fc7c0 20 add_watcher() i=1 2024-01-16T04:37:30.845 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.848+0000 7faf4c1fc7c0 20 add_watcher() i=2 2024-01-16T04:37:30.845 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.848+0000 7faf4c1fc7c0 20 add_watcher() i=3 2024-01-16T04:37:30.845 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.848+0000 7faf4c1fc7c0 20 add_watcher() i=4 2024-01-16T04:37:30.845 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.848+0000 7faf4c1fc7c0 20 add_watcher() i=5 2024-01-16T04:37:30.845 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.848+0000 7faf4c1fc7c0 20 add_watcher() i=6 2024-01-16T04:37:30.847 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.851+0000 7faf4c1fc7c0 20 add_watcher() i=7 2024-01-16T04:37:30.847 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.851+0000 7faf4c1fc7c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:37:30.848 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.852+0000 7fae83fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:37:30.849 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.852+0000 7faf4c1fc7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:37:30.849 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.852+0000 7faf4c1fc7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:37:30.852 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.856+0000 7faf4c1fc7c0 5 note: GC not initialized 2024-01-16T04:37:30.854 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.857+0000 7fae72ffd700 20 reqs_thread_entry: start 2024-01-16T04:37:30.854 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.857+0000 7fae70ff9700 20 reqs_thread_entry: start 2024-01-16T04:37:30.854 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.857+0000 7faf4c1fc7c0 20 init_complete bucket index max shards: 11 2024-01-16T04:37:30.854 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.857+0000 7faf4c1fc7c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:37:30.854 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.857+0000 7fae6affd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:38:01 2024 2024-01-16T04:37:30.854 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:37:30.854 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.857+0000 7fae6a7fc700 20 reqs_thread_entry: start 2024-01-16T04:37:30.855 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.857+0000 7faf4c1fc7c0 20 get_system_obj_state: rctx=0x7ffe16e81ba8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x5614fb725d50 s->prefetch_data=0 2024-01-16T04:37:30.855 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.857+0000 7faf4c1fc7c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-01-16T04:37:30.855 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.857+0000 7faf4c1fc7c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2024-01-16T04:37:30.855 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.857+0000 7faf4c1fc7c0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:37:30.855 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.857+0000 7faf4c1fc7c0 20 get_system_obj_state: rctx=0x7ffe16e81bb8 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x5614fb725d50 s->prefetch_data=0 2024-01-16T04:37:30.855 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.857+0000 7faf4c1fc7c0 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2024-01-16T04:37:30.855 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.858+0000 7faf4c1fc7c0 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2024-01-16T04:37:30.855 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.858+0000 7faf4c1fc7c0 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2024-01-16T04:37:30.855 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.858+0000 7faf4c1fc7c0 20 get_system_obj_state: rctx=0x7ffe16e81bb8 obj=default.rgw.meta:users.keys:PLSGDJEUQHITTOSCZDQL state=0x5614fb725d50 s->prefetch_data=0 2024-01-16T04:37:30.855 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.858+0000 7faf4c1fc7c0 10 cache get: name=default.rgw.meta+users.keys+PLSGDJEUQHITTOSCZDQL : miss 2024-01-16T04:37:30.856 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.858+0000 7faf4c1fc7c0 10 cache put: name=default.rgw.meta+users.keys+PLSGDJEUQHITTOSCZDQL info.flags=0x0 2024-01-16T04:37:30.856 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.858+0000 7faf4c1fc7c0 10 adding default.rgw.meta+users.keys+PLSGDJEUQHITTOSCZDQL to cache LRU end 2024-01-16T04:37:30.857 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.858+0000 7faf4c1fc7c0 20 get_system_obj_state: rctx=0x7ffe16e814d8 obj=default.rgw.meta:users.keys:PLSGDJEUQHITTOSCZDQL state=0x5614fb725d50 s->prefetch_data=0 2024-01-16T04:37:30.857 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.858+0000 7faf4c1fc7c0 10 cache get: name=default.rgw.meta+users.keys+PLSGDJEUQHITTOSCZDQL : hit (negative entry) 2024-01-16T04:37:30.857 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.858+0000 7faf4c1fc7c0 20 get_system_obj_state: rctx=0x7ffe16e81818 obj=default.rgw.meta:users.keys:PLSGDJEUQHITTOSCZDQL state=0x5614fb725d50 s->prefetch_data=0 2024-01-16T04:37:30.857 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.858+0000 7faf4c1fc7c0 10 cache get: name=default.rgw.meta+users.keys+PLSGDJEUQHITTOSCZDQL : hit (negative entry) 2024-01-16T04:37:30.857 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.859+0000 7faf4c1fc7c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-01-16T04:37:30.857 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.859+0000 7faf4c1fc7c0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:37:30.857 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.859+0000 7faf4c1fc7c0 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-01-16T04:37:30.857 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.859+0000 7faf35358700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084290 cookie 94648119239952 notifier 4226 bl.length()=714 2024-01-16T04:37:30.857 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.859+0000 7faf35358700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-01-16T04:37:30.857 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.859+0000 7faf35358700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:37:30.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.862+0000 7faf4c1fc7c0 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2024-01-16T04:37:30.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.862+0000 7faf4c1fc7c0 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2024-01-16T04:37:30.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.862+0000 7faf4c1fc7c0 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2024-01-16T04:37:30.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.862+0000 7faf35358700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084291 cookie 94648119247808 notifier 4226 bl.length()=224 2024-01-16T04:37:30.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.862+0000 7faf35358700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2024-01-16T04:37:30.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.862+0000 7faf35358700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2024-01-16T04:37:30.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.864+0000 7faf4c1fc7c0 10 cache put: name=default.rgw.meta+users.keys+PLSGDJEUQHITTOSCZDQL info.flags=0x7 2024-01-16T04:37:30.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.864+0000 7faf4c1fc7c0 10 moving default.rgw.meta+users.keys+PLSGDJEUQHITTOSCZDQL to cache LRU end 2024-01-16T04:37:30.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.864+0000 7faf4c1fc7c0 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:PLSGDJEUQHITTOSCZDQL, ofs0, ns] 2024-01-16T04:37:30.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.865+0000 7faf34b57700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084288 cookie 94648119226256 notifier 4226 bl.length()=201 2024-01-16T04:37:30.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.865+0000 7faf34b57700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+PLSGDJEUQHITTOSCZDQL info.flags=0x7 2024-01-16T04:37:30.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.865+0000 7faf34b57700 10 rgw watcher librados: moving default.rgw.meta+users.keys+PLSGDJEUQHITTOSCZDQL to cache LRU end 2024-01-16T04:37:30.862 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-01-16T04:37:30.862 INFO:teuthology.orchestra.run.smithi176.stdout: "user_id": "testx$tenanteduser.client.0", 2024-01-16T04:37:30.862 INFO:teuthology.orchestra.run.smithi176.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2024-01-16T04:37:30.862 INFO:teuthology.orchestra.run.smithi176.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2024-01-16T04:37:30.862 INFO:teuthology.orchestra.run.smithi176.stdout: "suspended": 0, 2024-01-16T04:37:30.862 INFO:teuthology.orchestra.run.smithi176.stdout: "max_buckets": 1000, 2024-01-16T04:37:30.862 INFO:teuthology.orchestra.run.smithi176.stdout: "subusers": [], 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: "keys": [ 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: "user": "testx$tenanteduser.client.0", 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: "access_key": "PLSGDJEUQHITTOSCZDQL", 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: "secret_key": "00xMWi+7SMUHJ790u01K/K8/ikNoNcTpp8tMYG5q8aEYTc2sntPwDw==" 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: "swift_keys": [], 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: "caps": [ 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "user-policy", 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: "perm": "*" 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-01-16T04:37:30.863 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-01-16T04:37:30.864 INFO:teuthology.orchestra.run.smithi176.stdout: "op_mask": "read, write, delete", 2024-01-16T04:37:30.864 INFO:teuthology.orchestra.run.smithi176.stdout: "default_placement": "", 2024-01-16T04:37:30.864 INFO:teuthology.orchestra.run.smithi176.stdout: "default_storage_class": "", 2024-01-16T04:37:30.864 INFO:teuthology.orchestra.run.smithi176.stdout: "placement_tags": [], 2024-01-16T04:37:30.864 INFO:teuthology.orchestra.run.smithi176.stdout: "bucket_quota": { 2024-01-16T04:37:30.864 INFO:teuthology.orchestra.run.smithi176.stdout: "enabled": false, 2024-01-16T04:37:30.864 INFO:teuthology.orchestra.run.smithi176.stdout: "check_on_raw": false, 2024-01-16T04:37:30.864 INFO:teuthology.orchestra.run.smithi176.stdout: "max_size": -1, 2024-01-16T04:37:30.865 INFO:teuthology.orchestra.run.smithi176.stdout: "max_size_kb": 0, 2024-01-16T04:37:30.865 INFO:teuthology.orchestra.run.smithi176.stdout: "max_objects": -1 2024-01-16T04:37:30.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-01-16T04:37:30.865 INFO:teuthology.orchestra.run.smithi176.stdout: "user_quota": { 2024-01-16T04:37:30.865 INFO:teuthology.orchestra.run.smithi176.stdout: "enabled": false, 2024-01-16T04:37:30.865 INFO:teuthology.orchestra.run.smithi176.stdout: "check_on_raw": false, 2024-01-16T04:37:30.865 INFO:teuthology.orchestra.run.smithi176.stdout: "max_size": -1, 2024-01-16T04:37:30.865 INFO:teuthology.orchestra.run.smithi176.stdout: "max_size_kb": 0, 2024-01-16T04:37:30.865 INFO:teuthology.orchestra.run.smithi176.stdout: "max_objects": -1 2024-01-16T04:37:30.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-01-16T04:37:30.865 INFO:teuthology.orchestra.run.smithi176.stdout: "temp_url_keys": [], 2024-01-16T04:37:30.866 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "rgw", 2024-01-16T04:37:30.866 INFO:teuthology.orchestra.run.smithi176.stdout: "mfa_ids": [] 2024-01-16T04:37:30.866 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-01-16T04:37:30.866 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:37:30.866 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.867+0000 7faf4c1fc7c0 20 remove_watcher() i=0 2024-01-16T04:37:30.866 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.867+0000 7faf4c1fc7c0 2 removed watcher, disabling cache 2024-01-16T04:37:30.866 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.869+0000 7faf4c1fc7c0 20 remove_watcher() i=1 2024-01-16T04:37:30.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.872+0000 7faf4c1fc7c0 20 remove_watcher() i=2 2024-01-16T04:37:30.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.873+0000 7faf4c1fc7c0 20 remove_watcher() i=3 2024-01-16T04:37:30.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.874+0000 7faf4c1fc7c0 20 remove_watcher() i=4 2024-01-16T04:37:30.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.874+0000 7faf4c1fc7c0 20 remove_watcher() i=5 2024-01-16T04:37:30.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.875+0000 7faf4c1fc7c0 20 remove_watcher() i=6 2024-01-16T04:37:30.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.876+0000 7faf4c1fc7c0 20 remove_watcher() i=7 2024-01-16T04:37:30.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.876+0000 7fae83fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:37:30.877 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 2444364063 --totp-seed PLOPWAHF5EFBZR2ROSKQPTSVV4C4Z7AODI43SROSQQO3Q6462R7KLINLEBFTWQOU --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-01-16T04:37:30.925 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:37:30.925 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:37:30.931 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.935+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.932 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.935+0000 7fb6f04be7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.933 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.935+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.933 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.936+0000 7fb6f04be7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.933 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.936+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.933 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.936+0000 7fb6f04be7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.933 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.936+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.933 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.936+0000 7fb6f04be7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.943+0000 7fb6f04be7c0 20 RGWRados::pool_iterate: got zonegroup_info.ddb5a393-8a31-4788-b116-e494d90f23e0 2024-01-16T04:37:30.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.943+0000 7fb6f04be7c0 20 RGWRados::pool_iterate: got zone_info.9cd49eee-6ce1-47e3-8681-41594a956139 2024-01-16T04:37:30.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.943+0000 7fb6f04be7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:37:30.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.943+0000 7fb6f04be7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:37:30.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.943+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.943+0000 7fb6f04be7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.943+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.944+0000 7fb6f04be7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:37:30.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.944+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.944+0000 7fb6f04be7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.944+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.944+0000 7fb6f04be7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:37:30.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.944+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.944+0000 7fb6f04be7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.944+0000 7fb6f04be7c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:37:30.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.944+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.944+0000 7fb6f04be7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.944+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.945+0000 7fb6f04be7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.945+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.945+0000 7fb6f04be7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.945+0000 7fb6f04be7c0 20 zonegroup default 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.945+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.945+0000 7fb6f04be7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.945+0000 7fb6f04be7c0 10 Cannot find current period zone using local zone 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.945+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.945+0000 7fb6f04be7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.945+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.945+0000 7fb6f04be7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.945+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.946+0000 7fb6f04be7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.946+0000 7fb6f04be7c0 20 zone default found 2024-01-16T04:37:30.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.946+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.946+0000 7fb6f04be7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.946+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.946+0000 7fb6f04be7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:37:30.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.946+0000 7fb6f04be7c0 20 started sync module instance, tier type = 2024-01-16T04:37:30.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.946+0000 7fb6f04be7c0 20 started zone id=9cd49eee-6ce1-47e3-8681-41594a956139 (name=default) with tier type = 2024-01-16T04:37:30.952 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.956+0000 7fb6f04be7c0 20 add_watcher() i=0 2024-01-16T04:37:30.952 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.956+0000 7fb6f04be7c0 20 add_watcher() i=1 2024-01-16T04:37:30.952 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.956+0000 7fb6f04be7c0 20 add_watcher() i=2 2024-01-16T04:37:30.952 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.956+0000 7fb6f04be7c0 20 add_watcher() i=3 2024-01-16T04:37:30.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.956+0000 7fb6f04be7c0 20 add_watcher() i=4 2024-01-16T04:37:30.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.956+0000 7fb6f04be7c0 20 add_watcher() i=5 2024-01-16T04:37:30.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.956+0000 7fb6f04be7c0 20 add_watcher() i=6 2024-01-16T04:37:30.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.957+0000 7fb6f04be7c0 20 add_watcher() i=7 2024-01-16T04:37:30.954 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.957+0000 7fb6f04be7c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:37:30.955 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.958+0000 7fb6acfe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:37:30.955 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.958+0000 7fb6f04be7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:37:30.955 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.959+0000 7fb6f04be7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:37:30.958 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.962+0000 7fb6f04be7c0 5 note: GC not initialized 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.963+0000 7fb61b7fe700 20 reqs_thread_entry: start 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.963+0000 7fb6197fa700 20 reqs_thread_entry: start 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.963+0000 7fb6f04be7c0 20 init_complete bucket index max shards: 11 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6f04be7c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6077fe700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:38:01 2024 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb606ffd700 20 reqs_thread_entry: start 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6f04be7c0 20 get_system_obj_state: rctx=0x7ffd9d1d7ca8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55b263b8dd50 s->prefetch_data=0 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6f04be7c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6f04be7c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6f04be7c0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6f04be7c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:37:30.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6f04be7c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:37:30.962 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6f04be7c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:37:30.962 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6f04be7c0 20 rados->read ofs=0 len=0 2024-01-16T04:37:30.962 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6f04be7c0 20 rados_obj.operate() r=0 bl.length=514 2024-01-16T04:37:30.962 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6f04be7c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x11 2024-01-16T04:37:30.962 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.964+0000 7fb6f04be7c0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:37:30.963 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.967+0000 7fb6f04be7c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-01-16T04:37:30.964 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.967+0000 7fb6f04be7c0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:37:30.964 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.967+0000 7fb6f04be7c0 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-01-16T04:37:30.964 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.967+0000 7fb6d961a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084292 cookie 94224666927568 notifier 4232 bl.length()=728 2024-01-16T04:37:30.964 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.967+0000 7fb6d961a700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-01-16T04:37:30.964 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.967+0000 7fb6d961a700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:37:30.965 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.969+0000 7fb6f04be7c0 20 remove_watcher() i=0 2024-01-16T04:37:30.965 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.969+0000 7fb6f04be7c0 2 removed watcher, disabling cache 2024-01-16T04:37:30.966 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.970+0000 7fb6f04be7c0 20 remove_watcher() i=1 2024-01-16T04:37:30.967 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.971+0000 7fb6f04be7c0 20 remove_watcher() i=2 2024-01-16T04:37:30.968 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.972+0000 7fb6f04be7c0 20 remove_watcher() i=3 2024-01-16T04:37:30.969 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.972+0000 7fb6f04be7c0 20 remove_watcher() i=4 2024-01-16T04:37:30.970 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.973+0000 7fb6f04be7c0 20 remove_watcher() i=5 2024-01-16T04:37:30.972 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.976+0000 7fb6f04be7c0 20 remove_watcher() i=6 2024-01-16T04:37:30.973 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.977+0000 7fb6f04be7c0 20 remove_watcher() i=7 2024-01-16T04:37:30.973 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:37:30.977+0000 7fb6acfe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:37:30.978 INFO:tasks.s3tests:Configuring s3-tests... 2024-01-16T04:37:30.978 DEBUG:teuthology.orchestra.run.smithi176:> cd /home/ubuntu/cephtest/s3-tests && ./bootstrap 2024-01-16T04:37:40.105 INFO:teuthology.orchestra.run.smithi176.stderr:./bootstrap: Missing required RPM packages: libevent-devel libffi-devel libxml2-devel libxslt-devel zlib-devel. 2024-01-16T04:37:40.944 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:04:11 ago on Tue 16 Jan 2024 04:33:29 AM UTC. 2024-01-16T04:37:42.679 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-01-16T04:37:42.681 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:37:42.681 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-01-16T04:37:42.681 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:37:42.681 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-01-16T04:37:42.681 INFO:teuthology.orchestra.run.smithi176.stdout: libevent-devel x86_64 2.1.8-5.el8 appstream 104 k 2024-01-16T04:37:42.681 INFO:teuthology.orchestra.run.smithi176.stdout: libffi-devel x86_64 3.1-24.el8 baseos 29 k 2024-01-16T04:37:42.681 INFO:teuthology.orchestra.run.smithi176.stdout: libxml2-devel x86_64 2.9.7-18.el8 appstream 1.1 M 2024-01-16T04:37:42.681 INFO:teuthology.orchestra.run.smithi176.stdout: libxslt-devel x86_64 1.1.32-6.el8 appstream 323 k 2024-01-16T04:37:42.681 INFO:teuthology.orchestra.run.smithi176.stdout: zlib-devel x86_64 1.2.11-25.el8 baseos 59 k 2024-01-16T04:37:42.681 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout: libxml2 x86_64 2.9.7-18.el8 baseos 697 k 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout: python3-libxml2 x86_64 2.9.7-18.el8 baseos 238 k 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout: zlib x86_64 1.2.11-25.el8 baseos 103 k 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout: cmake-filesystem x86_64 3.26.5-2.el8 appstream 52 k 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout: libgcrypt-devel x86_64 1.8.5-7.el8 baseos 150 k 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout: libgpg-error-devel x86_64 1.31-1.el8 baseos 64 k 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout: xz-devel x86_64 5.2.4-4.el8 baseos 63 k 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout:Install 9 Packages 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 3 Packages 2024-01-16T04:37:42.682 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:37:42.683 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 2.9 M 2024-01-16T04:37:42.683 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-01-16T04:37:43.349 INFO:teuthology.orchestra.run.smithi176.stdout:(1/12): cmake-filesystem-3.26.5-2.el8.x86_64.rp 194 kB/s | 52 kB 00:00 2024-01-16T04:37:43.365 INFO:teuthology.orchestra.run.smithi176.stdout:(2/12): libevent-devel-2.1.8-5.el8.x86_64.rpm 366 kB/s | 104 kB 00:00 2024-01-16T04:37:43.424 INFO:teuthology.orchestra.run.smithi176.stdout:(3/12): libffi-devel-3.1-24.el8.x86_64.rpm 499 kB/s | 29 kB 00:00 2024-01-16T04:37:43.491 INFO:teuthology.orchestra.run.smithi176.stdout:(4/12): libgcrypt-devel-1.8.5-7.el8.x86_64.rpm 2.2 MB/s | 150 kB 00:00 2024-01-16T04:37:43.516 INFO:teuthology.orchestra.run.smithi176.stdout:(5/12): libgpg-error-devel-1.31-1.el8.x86_64.rp 2.5 MB/s | 64 kB 00:00 2024-01-16T04:37:43.558 INFO:teuthology.orchestra.run.smithi176.stdout:(6/12): libxml2-devel-2.9.7-18.el8.x86_64.rpm 2.3 MB/s | 1.1 MB 00:00 2024-01-16T04:37:43.584 INFO:teuthology.orchestra.run.smithi176.stdout:(7/12): libxslt-devel-1.1.32-6.el8.x86_64.rpm 1.3 MB/s | 323 kB 00:00 2024-01-16T04:37:43.601 INFO:teuthology.orchestra.run.smithi176.stdout:(8/12): xz-devel-5.2.4-4.el8.x86_64.rpm 745 kB/s | 63 kB 00:00 2024-01-16T04:37:43.634 INFO:teuthology.orchestra.run.smithi176.stdout:(9/12): zlib-devel-1.2.11-25.el8.x86_64.rpm 778 kB/s | 59 kB 00:00 2024-01-16T04:37:43.659 INFO:teuthology.orchestra.run.smithi176.stdout:(10/12): python3-libxml2-2.9.7-18.el8.x86_64.rp 4.0 MB/s | 238 kB 00:00 2024-01-16T04:37:43.684 INFO:teuthology.orchestra.run.smithi176.stdout:(11/12): zlib-1.2.11-25.el8.x86_64.rpm 2.0 MB/s | 103 kB 00:00 2024-01-16T04:37:43.768 INFO:teuthology.orchestra.run.smithi176.stdout:(12/12): libxml2-2.9.7-18.el8.x86_64.rpm 3.7 MB/s | 697 kB 00:00 2024-01-16T04:37:43.768 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:37:43.768 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.7 MB/s | 2.9 MB 00:01 2024-01-16T04:37:43.883 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-01-16T04:37:44.022 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-01-16T04:37:44.022 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-01-16T04:37:44.161 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-01-16T04:37:44.164 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-01-16T04:37:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-01-16T04:37:44.915 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : zlib-1.2.11-25.el8.x86_64 1/15 2024-01-16T04:37:45.151 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : libxml2-2.9.7-18.el8.x86_64 2/15 2024-01-16T04:37:45.274 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 3/15 2024-01-16T04:37:45.297 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 3/15 2024-01-16T04:37:45.572 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgcrypt-devel-1.8.5-7.el8.x86_64 4/15 2024-01-16T04:37:45.585 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libgcrypt-devel-1.8.5-7.el8.x86_64 4/15 2024-01-16T04:37:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : zlib-devel-1.2.11-25.el8.x86_64 5/15 2024-01-16T04:37:45.956 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xz-devel-5.2.4-4.el8.x86_64 6/15 2024-01-16T04:37:46.399 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cmake-filesystem-3.26.5-2.el8.x86_64 7/15 2024-01-16T04:37:46.712 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libxml2-devel-2.9.7-18.el8.x86_64 8/15 2024-01-16T04:37:46.969 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 9/15 2024-01-16T04:37:47.212 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : python3-libxml2-2.9.7-18.el8.x86_64 10/15 2024-01-16T04:37:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libffi-devel-3.1-24.el8.x86_64 11/15 2024-01-16T04:37:47.368 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libffi-devel-3.1-24.el8.x86_64 11/15 2024-01-16T04:37:47.591 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 12/15 2024-01-16T04:37:47.775 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : python3-libxml2-2.9.7-16.el8.x86_64 13/15 2024-01-16T04:37:47.933 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : libxml2-2.9.7-16.el8.x86_64 14/15 2024-01-16T04:37:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : zlib-1.2.11-21.el8.x86_64 15/15 2024-01-16T04:37:48.462 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: zlib-1.2.11-21.el8.x86_64 15/15 2024-01-16T04:37:48.462 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cmake-filesystem-3.26.5-2.el8.x86_64 1/15 2024-01-16T04:37:48.462 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 2/15 2024-01-16T04:37:48.462 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libxml2-devel-2.9.7-18.el8.x86_64 3/15 2024-01-16T04:37:48.462 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 4/15 2024-01-16T04:37:48.462 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libffi-devel-3.1-24.el8.x86_64 5/15 2024-01-16T04:37:48.462 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgcrypt-devel-1.8.5-7.el8.x86_64 6/15 2024-01-16T04:37:48.463 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 7/15 2024-01-16T04:37:48.463 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xz-devel-5.2.4-4.el8.x86_64 8/15 2024-01-16T04:37:48.463 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : zlib-devel-1.2.11-25.el8.x86_64 9/15 2024-01-16T04:37:48.463 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libxml2-2.9.7-18.el8.x86_64 10/15 2024-01-16T04:37:48.463 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libxml2-2.9.7-16.el8.x86_64 11/15 2024-01-16T04:37:48.463 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-libxml2-2.9.7-18.el8.x86_64 12/15 2024-01-16T04:37:48.463 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-libxml2-2.9.7-16.el8.x86_64 13/15 2024-01-16T04:37:48.463 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : zlib-1.2.11-25.el8.x86_64 14/15 2024-01-16T04:37:49.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : zlib-1.2.11-21.el8.x86_64 15/15 2024-01-16T04:37:49.327 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:37:49.327 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2024-01-16T04:37:49.327 INFO:teuthology.orchestra.run.smithi176.stdout: libxml2-2.9.7-18.el8.x86_64 python3-libxml2-2.9.7-18.el8.x86_64 2024-01-16T04:37:49.328 INFO:teuthology.orchestra.run.smithi176.stdout: zlib-1.2.11-25.el8.x86_64 2024-01-16T04:37:49.328 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-01-16T04:37:49.328 INFO:teuthology.orchestra.run.smithi176.stdout: cmake-filesystem-3.26.5-2.el8.x86_64 libevent-devel-2.1.8-5.el8.x86_64 2024-01-16T04:37:49.328 INFO:teuthology.orchestra.run.smithi176.stdout: libffi-devel-3.1-24.el8.x86_64 libgcrypt-devel-1.8.5-7.el8.x86_64 2024-01-16T04:37:49.328 INFO:teuthology.orchestra.run.smithi176.stdout: libgpg-error-devel-1.31-1.el8.x86_64 libxml2-devel-2.9.7-18.el8.x86_64 2024-01-16T04:37:49.328 INFO:teuthology.orchestra.run.smithi176.stdout: libxslt-devel-1.1.32-6.el8.x86_64 xz-devel-5.2.4-4.el8.x86_64 2024-01-16T04:37:49.328 INFO:teuthology.orchestra.run.smithi176.stdout: zlib-devel-1.2.11-25.el8.x86_64 2024-01-16T04:37:49.328 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:37:49.328 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-01-16T04:37:49.910 INFO:teuthology.orchestra.run.smithi176.stdout:Already using interpreter /usr/bin/python3.6 2024-01-16T04:37:49.910 INFO:teuthology.orchestra.run.smithi176.stdout:Using base prefix '/usr' 2024-01-16T04:37:49.910 INFO:teuthology.orchestra.run.smithi176.stdout:New python executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python3.6 2024-01-16T04:37:49.910 INFO:teuthology.orchestra.run.smithi176.stdout:Also creating executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python 2024-01-16T04:37:53.118 INFO:teuthology.orchestra.run.smithi176.stdout:Installing setuptools, pip, wheel...done. 2024-01-16T04:37:53.449 INFO:teuthology.orchestra.run.smithi176.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.6/site-packages (21.3.1) 2024-01-16T04:37:54.761 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting setuptools==44.1.0 2024-01-16T04:37:54.844 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2024-01-16T04:37:55.022 INFO:teuthology.orchestra.run.smithi176.stdout:Installing collected packages: setuptools 2024-01-16T04:37:55.022 INFO:teuthology.orchestra.run.smithi176.stdout: Attempting uninstall: setuptools 2024-01-16T04:37:55.022 INFO:teuthology.orchestra.run.smithi176.stdout: Found existing installation: setuptools 59.6.0 2024-01-16T04:37:55.068 INFO:teuthology.orchestra.run.smithi176.stdout: Uninstalling setuptools-59.6.0: 2024-01-16T04:37:55.076 INFO:teuthology.orchestra.run.smithi176.stdout: Successfully uninstalled setuptools-59.6.0 2024-01-16T04:37:55.290 INFO:teuthology.orchestra.run.smithi176.stdout:Successfully installed setuptools-44.1.0 2024-01-16T04:37:55.926 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting PyYAML 2024-01-16T04:37:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB) 2024-01-16T04:37:56.238 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting nose>=1.0.0 2024-01-16T04:37:56.256 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2024-01-16T04:37:56.367 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting boto>=2.6.0 2024-01-16T04:37:56.400 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2024-01-16T04:37:57.424 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting boto3>=1.0.0 2024-01-16T04:37:57.447 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2024-01-16T04:37:58.582 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting botocore<1.28.0 2024-01-16T04:37:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2024-01-16T04:37:58.867 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting munch>=2.0.0 2024-01-16T04:37:58.884 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading munch-4.0.0-py2.py3-none-any.whl (9.9 kB) 2024-01-16T04:37:59.338 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting gevent>=1.0 2024-01-16T04:37:59.384 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading gevent-22.10.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB) 2024-01-16T04:37:59.625 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting isodate>=0.4.4 2024-01-16T04:37:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2024-01-16T04:37:59.766 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting requests>=2.23.0 2024-01-16T04:37:59.783 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-01-16T04:37:59.967 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting pytz>=2011k 2024-01-16T04:37:59.986 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) 2024-01-16T04:38:00.092 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting httplib2 2024-01-16T04:38:00.107 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading httplib2-0.22.0-py3-none-any.whl (96 kB) 2024-01-16T04:38:00.707 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting lxml 2024-01-16T04:38:00.731 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading lxml-5.1.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.1 MB) 2024-01-16T04:38:01.253 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2024-01-16T04:38:01.271 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2024-01-16T04:38:01.505 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting jmespath<2.0.0,>=0.7.1 2024-01-16T04:38:01.541 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2024-01-16T04:38:01.654 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting python-dateutil<3.0.0,>=2.1 2024-01-16T04:38:01.677 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2024-01-16T04:38:01.830 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting urllib3<1.27,>=1.25.4 2024-01-16T04:38:01.866 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2024-01-16T04:38:02.089 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting importlib-metadata>=1.7.0 2024-01-16T04:38:02.129 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB) 2024-01-16T04:38:02.726 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting greenlet>=2.0.0 2024-01-16T04:38:02.745 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading greenlet-2.0.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (564 kB) 2024-01-16T04:38:03.145 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting zope.interface 2024-01-16T04:38:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253 kB) 2024-01-16T04:38:03.203 INFO:teuthology.orchestra.run.smithi176.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (from gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2024-01-16T04:38:03.264 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting zope.event 2024-01-16T04:38:03.281 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB) 2024-01-16T04:38:03.362 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting six 2024-01-16T04:38:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2024-01-16T04:38:03.486 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting certifi>=2017.4.17 2024-01-16T04:38:03.504 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading certifi-2023.11.17-py3-none-any.whl (162 kB) 2024-01-16T04:38:03.608 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting idna<4,>=2.5 2024-01-16T04:38:03.625 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading idna-3.6-py3-none-any.whl (61 kB) 2024-01-16T04:38:03.955 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting charset-normalizer~=2.0.0 2024-01-16T04:38:03.973 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-01-16T04:38:04.128 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2024-01-16T04:38:04.148 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB) 2024-01-16T04:38:04.293 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting zipp>=0.5 2024-01-16T04:38:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB) 2024-01-16T04:38:04.417 INFO:teuthology.orchestra.run.smithi176.stdout:Collecting typing-extensions>=3.6.4 2024-01-16T04:38:04.435 INFO:teuthology.orchestra.run.smithi176.stdout: Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB) 2024-01-16T04:38:04.925 INFO:teuthology.orchestra.run.smithi176.stdout:Installing collected packages: six, urllib3, python-dateutil, jmespath, zipp, typing-extensions, botocore, zope.interface, zope.event, s3transfer, pyparsing, importlib-metadata, idna, greenlet, charset-normalizer, certifi, requests, PyYAML, pytz, nose, munch, lxml, isodate, httplib2, gevent, boto3, boto 2024-01-16T04:38:08.374 INFO:teuthology.orchestra.run.smithi176.stdout:Successfully installed PyYAML-6.0.1 boto-2.49.0 boto3-1.23.10 botocore-1.26.10 certifi-2023.11.17 charset-normalizer-2.0.12 gevent-22.10.2 greenlet-2.0.2 httplib2-0.22.0 idna-3.6 importlib-metadata-4.8.3 isodate-0.6.1 jmespath-0.10.0 lxml-5.1.0 munch-4.0.0 nose-1.3.7 pyparsing-3.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 requests-2.27.1 s3transfer-0.5.2 six-1.16.0 typing-extensions-4.1.1 urllib3-1.26.18 zipp-3.6.0 zope.event-4.6 zope.interface-5.5.2 2024-01-16T04:38:08.700 INFO:teuthology.orchestra.run.smithi176.stdout:running develop 2024-01-16T04:38:08.721 INFO:teuthology.orchestra.run.smithi176.stdout:running egg_info 2024-01-16T04:38:08.721 INFO:teuthology.orchestra.run.smithi176.stdout:creating s3tests.egg-info 2024-01-16T04:38:08.721 INFO:teuthology.orchestra.run.smithi176.stdout:writing s3tests.egg-info/PKG-INFO 2024-01-16T04:38:08.721 INFO:teuthology.orchestra.run.smithi176.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2024-01-16T04:38:08.721 INFO:teuthology.orchestra.run.smithi176.stdout:writing requirements to s3tests.egg-info/requires.txt 2024-01-16T04:38:08.721 INFO:teuthology.orchestra.run.smithi176.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2024-01-16T04:38:08.721 INFO:teuthology.orchestra.run.smithi176.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2024-01-16T04:38:08.731 INFO:teuthology.orchestra.run.smithi176.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2024-01-16T04:38:08.732 INFO:teuthology.orchestra.run.smithi176.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2024-01-16T04:38:08.738 INFO:teuthology.orchestra.run.smithi176.stdout:running build_ext 2024-01-16T04:38:08.738 INFO:teuthology.orchestra.run.smithi176.stdout:Creating /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/s3tests.egg-link (link to .) 2024-01-16T04:38:08.739 INFO:teuthology.orchestra.run.smithi176.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2024-01-16T04:38:08.739 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.739 INFO:teuthology.orchestra.run.smithi176.stdout:Installed /home/ubuntu/cephtest/s3-tests 2024-01-16T04:38:08.740 INFO:teuthology.orchestra.run.smithi176.stdout:Processing dependencies for s3tests==0.0.1 2024-01-16T04:38:08.849 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for isodate==0.6.1 2024-01-16T04:38:08.849 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: isodate 0.6.1 2024-01-16T04:38:08.849 INFO:teuthology.orchestra.run.smithi176.stdout:Adding isodate 0.6.1 to easy-install.pth file 2024-01-16T04:38:08.849 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.849 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.849 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for gevent==22.10.2 2024-01-16T04:38:08.849 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: gevent 22.10.2 2024-01-16T04:38:08.849 INFO:teuthology.orchestra.run.smithi176.stdout:Adding gevent 22.10.2 to easy-install.pth file 2024-01-16T04:38:08.849 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.849 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.850 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for munch==4.0.0 2024-01-16T04:38:08.850 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: munch 4.0.0 2024-01-16T04:38:08.850 INFO:teuthology.orchestra.run.smithi176.stdout:Adding munch 4.0.0 to easy-install.pth file 2024-01-16T04:38:08.851 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.851 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.851 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for PyYAML==6.0.1 2024-01-16T04:38:08.851 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: PyYAML 6.0.1 2024-01-16T04:38:08.851 INFO:teuthology.orchestra.run.smithi176.stdout:Adding PyYAML 6.0.1 to easy-install.pth file 2024-01-16T04:38:08.851 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.851 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.852 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for boto3==1.23.10 2024-01-16T04:38:08.852 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: boto3 1.23.10 2024-01-16T04:38:08.852 INFO:teuthology.orchestra.run.smithi176.stdout:Adding boto3 1.23.10 to easy-install.pth file 2024-01-16T04:38:08.852 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.852 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.852 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for boto==2.49.0 2024-01-16T04:38:08.852 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: boto 2.49.0 2024-01-16T04:38:08.853 INFO:teuthology.orchestra.run.smithi176.stdout:Adding boto 2.49.0 to easy-install.pth file 2024-01-16T04:38:08.853 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.853 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.854 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for six==1.16.0 2024-01-16T04:38:08.854 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: six 1.16.0 2024-01-16T04:38:08.854 INFO:teuthology.orchestra.run.smithi176.stdout:Adding six 1.16.0 to easy-install.pth file 2024-01-16T04:38:08.854 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.854 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.854 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for setuptools==44.1.0 2024-01-16T04:38:08.854 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: setuptools 44.1.0 2024-01-16T04:38:08.854 INFO:teuthology.orchestra.run.smithi176.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2024-01-16T04:38:08.855 INFO:teuthology.orchestra.run.smithi176.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2024-01-16T04:38:08.855 INFO:teuthology.orchestra.run.smithi176.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2024-01-16T04:38:08.855 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.855 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.856 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for zope.event==4.6 2024-01-16T04:38:08.856 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: zope.event 4.6 2024-01-16T04:38:08.856 INFO:teuthology.orchestra.run.smithi176.stdout:Adding zope.event 4.6 to easy-install.pth file 2024-01-16T04:38:08.856 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.856 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.857 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for greenlet==2.0.2 2024-01-16T04:38:08.857 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: greenlet 2.0.2 2024-01-16T04:38:08.857 INFO:teuthology.orchestra.run.smithi176.stdout:Adding greenlet 2.0.2 to easy-install.pth file 2024-01-16T04:38:08.857 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.857 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.857 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for zope.interface==5.5.2 2024-01-16T04:38:08.857 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: zope.interface 5.5.2 2024-01-16T04:38:08.858 INFO:teuthology.orchestra.run.smithi176.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2024-01-16T04:38:08.858 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.858 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.859 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for importlib-metadata==4.8.3 2024-01-16T04:38:08.859 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: importlib-metadata 4.8.3 2024-01-16T04:38:08.859 INFO:teuthology.orchestra.run.smithi176.stdout:Adding importlib-metadata 4.8.3 to easy-install.pth file 2024-01-16T04:38:08.859 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.859 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.859 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for s3transfer==0.5.2 2024-01-16T04:38:08.859 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: s3transfer 0.5.2 2024-01-16T04:38:08.859 INFO:teuthology.orchestra.run.smithi176.stdout:Adding s3transfer 0.5.2 to easy-install.pth file 2024-01-16T04:38:08.859 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.859 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.860 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for botocore==1.26.10 2024-01-16T04:38:08.860 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: botocore 1.26.10 2024-01-16T04:38:08.861 INFO:teuthology.orchestra.run.smithi176.stdout:Adding botocore 1.26.10 to easy-install.pth file 2024-01-16T04:38:08.861 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.861 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.861 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for jmespath==0.10.0 2024-01-16T04:38:08.861 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: jmespath 0.10.0 2024-01-16T04:38:08.861 INFO:teuthology.orchestra.run.smithi176.stdout:Adding jmespath 0.10.0 to easy-install.pth file 2024-01-16T04:38:08.861 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.861 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.862 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for typing-extensions==4.1.1 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: typing-extensions 4.1.1 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout:Adding typing-extensions 4.1.1 to easy-install.pth file 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for zipp==3.6.0 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: zipp 3.6.0 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout:Adding zipp 3.6.0 to easy-install.pth file 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for python-dateutil==2.8.2 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: python-dateutil 2.8.2 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.863 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.864 INFO:teuthology.orchestra.run.smithi176.stdout:Searching for urllib3==1.26.18 2024-01-16T04:38:08.864 INFO:teuthology.orchestra.run.smithi176.stdout:Best match: urllib3 1.26.18 2024-01-16T04:38:08.864 INFO:teuthology.orchestra.run.smithi176.stdout:Adding urllib3 1.26.18 to easy-install.pth file 2024-01-16T04:38:08.864 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:38:08.865 INFO:teuthology.orchestra.run.smithi176.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:38:08.865 INFO:teuthology.orchestra.run.smithi176.stdout:Finished processing dependencies for s3tests==0.0.1 2024-01-16T04:38:08.882 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:38:08.882 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2024-01-16T04:38:08.900 INFO:tasks.s3tests:Configuring boto... 2024-01-16T04:38:08.901 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-01-16T04:38:08.901 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/boto.cfg 2024-01-16T04:38:08.958 DEBUG:teuthology.orchestra.run.smithi176:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain' 2024-01-16T04:38:09.379 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2024-01-16T04:38:09.430 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2024-01-16T04:38:09.488 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2024-01-16T04:38:09.540 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2024-01-16T04:38:09.597 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2024-01-16T04:38:09.609 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2024-01-16T04:38:09.674 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2024-01-16T04:38:09.687 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2024-01-16T04:38:09.729 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2024-01-16T04:38:09.772 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2024-01-16T04:38:09.825 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2024-01-16T04:38:09.881 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2024-01-16T04:38:09.932 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2024-01-16T04:38:09.984 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2024-01-16T04:38:09.985 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2024-01-16T04:38:10.030 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2024-01-16T04:38:10.030 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2024-01-16T04:38:10.030 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2024-01-16T04:38:10.030 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2024-01-16T04:38:10.030 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2024-01-16T04:38:10.030 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2024-01-16T04:38:10.030 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2024-01-16T04:38:10.031 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2024-01-16T04:38:10.032 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2024-01-16T04:38:10.033 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2024-01-16T04:38:10.033 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2024-01-16T04:38:10.033 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2024-01-16T04:38:10.173 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2024-01-16T04:38:10.173 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_lifecycle_transition ... SKIP 2024-01-16T04:38:10.173 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2024-01-16T04:38:10.173 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2024-01-16T04:38:10.173 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2024-01-16T04:38:10.173 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2024-01-16T04:38:10.174 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2024-01-16T04:38:10.174 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2024-01-16T04:38:10.174 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2024-01-16T04:38:10.174 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2024-01-16T04:38:10.275 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2024-01-16T04:38:10.323 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2024-01-16T04:38:11.629 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2024-01-16T04:38:11.720 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2024-01-16T04:38:11.793 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2024-01-16T04:38:11.853 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_enc ... ok 2024-01-16T04:38:11.874 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2024-01-16T04:38:11.966 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2024-01-16T04:38:11.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2024-01-16T04:38:11.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2024-01-16T04:38:11.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2024-01-16T04:38:11.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2024-01-16T04:38:11.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2024-01-16T04:38:11.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2024-01-16T04:38:11.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2024-01-16T04:38:11.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2024-01-16T04:38:11.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2024-01-16T04:38:11.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2024-01-16T04:38:11.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2024-01-16T04:38:11.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_pubilc_errordoc ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2024-01-16T04:38:11.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2024-01-16T04:38:11.969 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2024-01-16T04:38:11.969 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2024-01-16T04:38:11.969 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2024-01-16T04:38:11.969 INFO:teuthology.orchestra.run.smithi176.stderr:SKIP 2024-01-16T04:38:15.343 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests.functional.test_utils.test_generate ... ok 2024-01-16T04:38:15.343 INFO:teuthology.orchestra.run.smithi176.stderr:/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2024-01-16T04:38:15.343 INFO:teuthology.orchestra.run.smithi176.stderr: warnings.warn(warning, PythonDeprecationWarning) 2024-01-16T04:38:15.367 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2024-01-16T04:38:27.896 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2024-01-16T04:38:27.931 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2024-01-16T04:38:27.982 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2024-01-16T04:38:28.023 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2024-01-16T04:38:28.062 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2024-01-16T04:38:28.103 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2024-01-16T04:38:28.155 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2024-01-16T04:38:28.201 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2024-01-16T04:38:28.231 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2024-01-16T04:38:28.262 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2024-01-16T04:38:28.287 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2024-01-16T04:38:28.303 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2024-01-16T04:38:28.318 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2024-01-16T04:38:28.327 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2024-01-16T04:38:28.352 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2024-01-16T04:38:28.386 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2024-01-16T04:38:28.427 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2024-01-16T04:38:28.457 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2024-01-16T04:38:28.485 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2024-01-16T04:38:28.515 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2024-01-16T04:38:28.540 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2024-01-16T04:38:28.564 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2024-01-16T04:38:28.581 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2024-01-16T04:38:28.595 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2024-01-16T04:38:28.603 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2024-01-16T04:38:28.611 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2024-01-16T04:38:28.620 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2024-01-16T04:38:28.627 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2024-01-16T04:38:28.634 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2024-01-16T04:38:28.713 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2024-01-16T04:38:28.810 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2024-01-16T04:38:28.911 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2024-01-16T04:38:29.018 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2024-01-16T04:38:29.105 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2024-01-16T04:38:29.161 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2024-01-16T04:38:29.254 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2024-01-16T04:38:29.315 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2024-01-16T04:38:29.368 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2024-01-16T04:38:29.498 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2024-01-16T04:38:29.625 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2024-01-16T04:38:29.659 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2024-01-16T04:38:29.694 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2024-01-16T04:38:29.758 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2024-01-16T04:38:29.823 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2024-01-16T04:38:29.949 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2024-01-16T04:38:30.098 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2024-01-16T04:38:30.158 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2024-01-16T04:38:30.217 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2024-01-16T04:38:30.270 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2024-01-16T04:38:30.328 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2024-01-16T04:38:30.389 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2024-01-16T04:38:30.455 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2024-01-16T04:38:30.513 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2024-01-16T04:38:30.576 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2024-01-16T04:38:30.636 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2024-01-16T04:38:30.696 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2024-01-16T04:38:30.751 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2024-01-16T04:38:30.810 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2024-01-16T04:38:30.893 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2024-01-16T04:38:30.956 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2024-01-16T04:38:31.005 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2024-01-16T04:38:31.066 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2024-01-16T04:38:31.138 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2024-01-16T04:38:38.978 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2024-01-16T04:38:39.062 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2024-01-16T04:38:39.109 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2024-01-16T04:38:39.156 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2024-01-16T04:38:39.221 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2024-01-16T04:38:39.274 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2024-01-16T04:38:39.329 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2024-01-16T04:38:39.378 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2024-01-16T04:38:39.427 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2024-01-16T04:38:39.475 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2024-01-16T04:38:39.532 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2024-01-16T04:38:39.590 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2024-01-16T04:38:39.640 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2024-01-16T04:38:39.695 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2024-01-16T04:38:39.768 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2024-01-16T04:38:39.832 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2024-01-16T04:38:39.887 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2024-01-16T04:38:39.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2024-01-16T04:38:40.034 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2024-01-16T04:38:40.095 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2024-01-16T04:38:40.155 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2024-01-16T04:38:40.209 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2024-01-16T04:38:40.271 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2024-01-16T04:38:40.384 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2024-01-16T04:38:40.485 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2024-01-16T04:38:40.539 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2024-01-16T04:38:40.597 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2024-01-16T04:38:40.654 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2024-01-16T04:38:40.715 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2024-01-16T04:38:40.983 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_account_usage ... ok 2024-01-16T04:38:41.018 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_head_bucket_usage ... ok 2024-01-16T04:38:41.459 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2024-01-16T04:38:41.867 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2024-01-16T04:38:41.931 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2024-01-16T04:38:41.986 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2024-01-16T04:38:42.044 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2024-01-16T04:38:42.101 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2024-01-16T04:38:42.214 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken ... ok 2024-01-16T04:38:42.319 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_both_continuationtoken_startafter ... ok 2024-01-16T04:38:42.377 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2024-01-16T04:38:42.448 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2024-01-16T04:38:42.520 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2024-01-16T04:38:42.576 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2024-01-16T04:38:42.658 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2024-01-16T04:38:42.722 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2024-01-16T04:38:42.962 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2024-01-16T04:38:43.269 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2024-01-16T04:38:43.307 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2024-01-16T04:38:43.346 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2024-01-16T04:38:43.373 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2024-01-16T04:38:43.400 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2024-01-16T04:38:43.409 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2024-01-16T04:38:43.419 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2024-01-16T04:38:43.427 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2024-01-16T04:38:43.465 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2024-01-16T04:38:43.624 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2024-01-16T04:38:43.633 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2024-01-16T04:38:43.728 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2024-01-16T04:38:43.756 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_read_not_exist ... ok 2024-01-16T04:38:43.782 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2024-01-16T04:38:44.008 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2024-01-16T04:38:44.243 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2024-01-16T04:38:44.275 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2024-01-16T04:38:44.303 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2024-01-16T04:38:44.335 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2024-01-16T04:38:44.368 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2024-01-16T04:38:44.536 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2024-01-16T04:38:44.570 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2024-01-16T04:38:44.606 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2024-01-16T04:38:44.654 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2024-01-16T04:38:44.693 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2024-01-16T04:38:44.775 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2024-01-16T04:38:44.848 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2024-01-16T04:38:44.888 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2024-01-16T04:38:44.962 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2024-01-16T04:38:44.982 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2024-01-16T04:38:45.005 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2024-01-16T04:38:45.026 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2024-01-16T04:38:45.102 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2024-01-16T04:38:45.133 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2024-01-16T04:38:45.160 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2024-01-16T04:38:45.189 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2024-01-16T04:38:45.223 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2024-01-16T04:38:45.296 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2024-01-16T04:38:45.326 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2024-01-16T04:38:45.355 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2024-01-16T04:38:45.383 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2024-01-16T04:38:45.407 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2024-01-16T04:38:45.429 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2024-01-16T04:38:45.453 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2024-01-16T04:38:45.485 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2024-01-16T04:38:45.507 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2024-01-16T04:38:45.538 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2024-01-16T04:38:45.569 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2024-01-16T04:38:45.598 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2024-01-16T04:38:45.627 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2024-01-16T04:38:45.658 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2024-01-16T04:38:45.684 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2024-01-16T04:38:45.708 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2024-01-16T04:38:45.732 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2024-01-16T04:38:45.756 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2024-01-16T04:38:45.781 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2024-01-16T04:38:45.805 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2024-01-16T04:38:45.877 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2024-01-16T04:38:45.960 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2024-01-16T04:38:45.997 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2024-01-16T04:38:46.073 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2024-01-16T04:38:46.174 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2024-01-16T04:38:47.216 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2024-01-16T04:38:47.299 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2024-01-16T04:38:47.379 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2024-01-16T04:38:47.511 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2024-01-16T04:38:47.681 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2024-01-16T04:38:47.803 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2024-01-16T04:38:47.916 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2024-01-16T04:38:48.038 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2024-01-16T04:38:48.203 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2024-01-16T04:38:48.283 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2024-01-16T04:38:48.449 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2024-01-16T04:38:48.484 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2024-01-16T04:38:48.556 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2024-01-16T04:38:48.637 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2024-01-16T04:38:48.689 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2024-01-16T04:38:48.721 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2024-01-16T04:38:48.730 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_notexist ... ok 2024-01-16T04:38:48.792 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2024-01-16T04:38:48.823 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2024-01-16T04:38:48.856 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2024-01-16T04:38:48.889 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2024-01-16T04:38:48.932 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2024-01-16T04:38:49.000 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2024-01-16T04:38:49.035 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2024-01-16T04:38:49.150 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2024-01-16T04:38:49.228 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2024-01-16T04:38:49.258 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2024-01-16T04:38:49.294 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired_tenant ... ok 2024-01-16T04:38:49.331 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2024-01-16T04:38:49.366 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2024-01-16T04:38:49.403 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2024-01-16T04:38:49.446 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2024-01-16T04:38:49.488 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2024-01-16T04:38:49.517 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2024-01-16T04:38:49.553 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2024-01-16T04:38:49.563 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2024-01-16T04:38:49.573 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2024-01-16T04:38:49.582 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2024-01-16T04:38:49.594 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2024-01-16T04:38:49.619 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2024-01-16T04:38:49.686 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2024-01-16T04:38:49.716 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2024-01-16T04:38:49.746 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl ... FAIL 2024-01-16T04:38:49.772 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl ... FAIL 2024-01-16T04:38:49.795 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2024-01-16T04:38:49.860 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2024-01-16T04:38:49.972 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2024-01-16T04:38:50.035 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2024-01-16T04:38:50.097 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2024-01-16T04:38:50.177 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2024-01-16T04:38:50.258 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2024-01-16T04:38:50.387 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2024-01-16T04:38:50.460 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2024-01-16T04:38:50.538 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2024-01-16T04:38:50.661 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2024-01-16T04:38:50.787 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2024-01-16T04:38:50.876 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2024-01-16T04:38:50.961 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2024-01-16T04:38:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2024-01-16T04:38:51.113 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2024-01-16T04:38:51.243 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2024-01-16T04:38:51.365 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2024-01-16T04:38:51.496 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2024-01-16T04:38:51.624 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2024-01-16T04:38:51.746 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2024-01-16T04:38:51.860 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2024-01-16T04:38:51.978 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2024-01-16T04:38:52.097 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2024-01-16T04:38:52.210 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2024-01-16T04:38:52.245 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2024-01-16T04:38:52.357 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2024-01-16T04:38:52.430 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2024-01-16T04:38:52.515 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2024-01-16T04:38:52.593 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2024-01-16T04:38:52.631 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_not_exist ... ok 2024-01-16T04:38:52.761 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2024-01-16T04:38:52.994 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2024-01-16T04:38:53.203 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2024-01-16T04:38:53.413 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2024-01-16T04:38:53.624 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2024-01-16T04:38:53.835 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2024-01-16T04:38:54.051 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2024-01-16T04:38:54.263 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2024-01-16T04:38:54.481 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2024-01-16T04:38:54.731 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2024-01-16T04:38:54.904 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2024-01-16T04:38:55.078 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2024-01-16T04:38:55.254 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2024-01-16T04:38:55.441 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2024-01-16T04:38:55.447 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2024-01-16T04:38:55.455 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2024-01-16T04:38:55.463 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2024-01-16T04:38:55.712 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2024-01-16T04:38:55.958 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2024-01-16T04:38:55.978 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2024-01-16T04:38:56.001 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2024-01-16T04:38:56.065 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2024-01-16T04:38:56.732 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2024-01-16T04:38:56.810 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2024-01-16T04:38:56.911 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2024-01-16T04:38:57.044 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2024-01-16T04:38:57.202 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2024-01-16T04:38:57.284 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2024-01-16T04:38:57.367 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2024-01-16T04:38:57.523 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2024-01-16T04:38:57.575 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2024-01-16T04:38:57.694 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2024-01-16T04:38:57.835 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2024-01-16T04:38:58.070 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2024-01-16T04:38:58.263 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2024-01-16T04:38:58.296 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2024-01-16T04:38:58.324 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2024-01-16T04:38:58.921 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2024-01-16T04:38:59.084 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2024-01-16T04:39:00.208 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2024-01-16T04:39:00.246 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2024-01-16T04:39:00.332 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2024-01-16T04:39:00.630 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2024-01-16T04:39:00.726 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2024-01-16T04:39:00.955 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2024-01-16T04:39:01.915 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2024-01-16T04:39:02.671 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2024-01-16T04:39:04.284 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2024-01-16T04:39:08.678 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2024-01-16T04:39:09.556 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2024-01-16T04:39:11.021 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2024-01-16T04:39:11.176 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2024-01-16T04:39:11.533 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2024-01-16T04:39:11.879 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2024-01-16T04:39:12.096 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2024-01-16T04:39:12.176 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2024-01-16T04:39:12.502 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2024-01-16T04:39:12.595 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2024-01-16T04:39:12.688 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2024-01-16T04:39:12.715 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2024-01-16T04:39:12.837 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2024-01-16T04:39:15.954 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2024-01-16T04:39:19.001 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2024-01-16T04:39:22.046 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2024-01-16T04:39:22.092 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_cors_presigned_get_object ... ok 2024-01-16T04:39:22.137 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_cors_presigned_get_object_tenant ... ok 2024-01-16T04:39:22.267 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_set_bucket_tagging ... ok 2024-01-16T04:39:22.364 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2024-01-16T04:39:22.542 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2024-01-16T04:39:22.823 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2024-01-16T04:39:22.970 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2024-01-16T04:39:23.230 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2024-01-16T04:39:23.628 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2024-01-16T04:39:23.753 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2024-01-16T04:39:24.018 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2024-01-16T04:39:24.483 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2024-01-16T04:39:24.616 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2024-01-16T04:39:24.787 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2024-01-16T04:39:25.089 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2024-01-16T04:39:25.164 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2024-01-16T04:39:30.888 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2024-01-16T04:39:30.967 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2024-01-16T04:39:31.048 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2024-01-16T04:39:31.129 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2024-01-16T04:39:31.203 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2024-01-16T04:39:31.467 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2024-01-16T04:39:39.198 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2024-01-16T04:39:39.762 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2024-01-16T04:39:39.968 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2024-01-16T04:39:40.275 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2024-01-16T04:39:40.626 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2024-01-16T04:39:43.632 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2024-01-16T04:39:47.228 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2024-01-16T04:40:01.646 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2024-01-16T04:40:03.149 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2024-01-16T04:40:03.829 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2024-01-16T04:40:04.816 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2024-01-16T04:40:05.232 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2024-01-16T04:40:05.686 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2024-01-16T04:40:05.821 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2024-01-16T04:40:06.290 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2024-01-16T04:40:06.670 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2024-01-16T04:40:07.098 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... ok 2024-01-16T04:40:07.133 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2024-01-16T04:40:07.212 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2024-01-16T04:40:07.293 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2024-01-16T04:40:07.329 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2024-01-16T04:40:07.364 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2024-01-16T04:40:07.481 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2024-01-16T04:40:07.519 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2024-01-16T04:40:07.555 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2024-01-16T04:40:07.618 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2024-01-16T04:40:07.654 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2024-01-16T04:40:07.686 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2024-01-16T04:40:07.720 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2024-01-16T04:40:07.760 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2024-01-16T04:40:07.839 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2024-01-16T04:40:07.918 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2024-01-16T04:40:07.977 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2024-01-16T04:40:08.053 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2024-01-16T04:40:08.096 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2024-01-16T04:40:08.180 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2024-01-16T04:40:08.263 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2024-01-16T04:40:08.364 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2024-01-16T04:40:08.438 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2024-01-16T04:40:08.516 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2024-01-16T04:40:08.588 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2024-01-16T04:40:09.471 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2024-01-16T04:40:10.077 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2024-01-16T04:40:10.114 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2024-01-16T04:40:10.164 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2024-01-16T04:40:10.247 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2024-01-16T04:40:10.322 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2024-01-16T04:40:10.397 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2024-01-16T04:40:11.299 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2024-01-16T04:40:11.820 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2024-01-16T04:40:12.325 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2024-01-16T04:40:12.363 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2024-01-16T04:40:12.443 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2024-01-16T04:40:12.524 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2024-01-16T04:40:12.578 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2024-01-16T04:40:12.655 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2024-01-16T04:40:12.738 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2024-01-16T04:40:12.788 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2024-01-16T04:40:12.837 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2024-01-16T04:40:12.900 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2024-01-16T04:40:13.035 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2024-01-16T04:40:13.129 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2024-01-16T04:40:13.262 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2024-01-16T04:40:13.434 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2024-01-16T04:40:13.562 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2024-01-16T04:40:13.732 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2024-01-16T04:40:13.904 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2024-01-16T04:40:14.076 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2024-01-16T04:40:14.246 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2024-01-16T04:40:14.421 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2024-01-16T04:40:14.645 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2024-01-16T04:40:14.862 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2024-01-16T04:40:14.979 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2024-01-16T04:40:15.020 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2024-01-16T04:40:15.140 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2024-01-16T04:40:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2024-01-16T04:40:15.476 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2024-01-16T04:40:15.661 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2024-01-16T04:40:15.934 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2024-01-16T04:40:17.619 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2024-01-16T04:40:17.752 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2024-01-16T04:40:17.966 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2024-01-16T04:40:18.142 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2024-01-16T04:40:18.425 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2024-01-16T04:40:18.607 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2024-01-16T04:40:18.698 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2024-01-16T04:40:18.868 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2024-01-16T04:40:19.083 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2024-01-16T04:40:19.163 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2024-01-16T04:40:19.236 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2024-01-16T04:40:19.304 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2024-01-16T04:40:19.378 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2024-01-16T04:40:19.523 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2024-01-16T04:40:19.592 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2024-01-16T04:40:19.655 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2024-01-16T04:40:19.728 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2024-01-16T04:40:19.793 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2024-01-16T04:40:19.856 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2024-01-16T04:40:19.932 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2024-01-16T04:40:20.066 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2024-01-16T04:40:20.169 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2024-01-16T04:40:20.244 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2024-01-16T04:40:20.362 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2024-01-16T04:40:20.475 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2024-01-16T04:40:20.593 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2024-01-16T04:40:20.693 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2024-01-16T04:40:20.806 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2024-01-16T04:40:20.904 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2024-01-16T04:40:20.950 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2024-01-16T04:40:21.027 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2024-01-16T04:40:21.114 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2024-01-16T04:40:21.247 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2024-01-16T04:40:21.324 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2024-01-16T04:40:21.427 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2024-01-16T04:40:21.479 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2024-01-16T04:40:21.578 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2024-01-16T04:40:21.689 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2024-01-16T04:40:21.836 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifmatch_good ... ok 2024-01-16T04:40:21.980 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifnonematch_failed ... ok 2024-01-16T04:40:22.005 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_user_policy ... ok 2024-01-16T04:40:22.037 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2024-01-16T04:40:22.127 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2024-01-16T04:40:22.213 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2024-01-16T04:40:22.302 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2024-01-16T04:40:22.397 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2024-01-16T04:40:22.479 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2024-01-16T04:40:22.507 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2024-01-16T04:40:22.610 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2024-01-16T04:40:22.822 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2024-01-16T04:40:22.987 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2024-01-16T04:40:23.068 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2024-01-16T04:40:23.256 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2024-01-16T04:41:53.862 INFO:teuthology.orchestra.run.smithi176.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2024-01-16T04:41:53.862 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:53.862 INFO:teuthology.orchestra.run.smithi176.stderr:====================================================================== 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr:FAIL: s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr:---------------------------------------------------------------------- 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr:Traceback (most recent call last): 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr: self.test(*self.arg) 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 4439, in test_bucket_recreate_overwrite_acl 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr: e = assert_raises(ClientError, client.create_bucket, Bucket=bucket_name) 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/utils.py", line 21, in assert_raises 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr: raise AssertionError("%s not raised" % excName) 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr:AssertionError: ClientError not raised 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr:-------------------- >> begin captured logging << -------------------- 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2024-01-16T04:41:53.863 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f7678ac4598> 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:41:53.864 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-q85zleljx4aymwl-211', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-acl': 'public-read', 'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-532.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi176.front.sepia.ceph.com:80/test-client.0-q85zleljx4aymwl-211', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-q85zleljx4aymwl-211'}}} 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:PUT 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:/test-client.0-q85zleljx4aymwl-211 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:host:smithi176.front.sepia.ceph.com 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:x-amz-acl:public-read 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr:x-amz-date:20240116T043849Z 2024-01-16T04:41:53.865 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:53.866 INFO:teuthology.orchestra.run.smithi176.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2024-01-16T04:41:53.866 INFO:teuthology.orchestra.run.smithi176.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:41:53.866 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: StringToSign: 2024-01-16T04:41:53.866 INFO:teuthology.orchestra.run.smithi176.stderr:AWS4-HMAC-SHA256 2024-01-16T04:41:53.866 INFO:teuthology.orchestra.run.smithi176.stderr:20240116T043849Z 2024-01-16T04:41:53.866 INFO:teuthology.orchestra.run.smithi176.stderr:20240116/us-east-1/s3/aws4_request 2024-01-16T04:41:53.866 INFO:teuthology.orchestra.run.smithi176.stderr:7a83728efedc2fa9886d21d3a2fd989641a184cc5d16e17afa2b760d1c33370e 2024-01-16T04:41:53.866 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: Signature: 2024-01-16T04:41:53.866 INFO:teuthology.orchestra.run.smithi176.stderr:1d50c0cbb651f089d1bfb87884f434e126de01d2188495cdc62b14f48a8c114e 2024-01-16T04:41:53.866 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi176.front.sepia.ceph.com:80 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:urllib3.connectionpool: DEBUG: http://smithi176.front.sepia.ceph.com:80 "PUT /test-client.0-q85zleljx4aymwl-211 HTTP/1.1" 200 0 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx0000069df4f786039572c-0065a60859-1064-default', 'Content-Length': '0', 'Date': 'Tue, 16 Jan 2024 04:38:49 GMT'} 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.parsers: DEBUG: Response body: 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:b'' 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:41:53.867 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:41:53.868 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.868 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:41:53.868 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:41:53.868 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-q85zleljx4aymwl-211', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-532.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi176.front.sepia.ceph.com:80/test-client.0-q85zleljx4aymwl-211', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-q85zleljx4aymwl-211'}}} 2024-01-16T04:41:53.868 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.869 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.869 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-01-16T04:41:53.869 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.869 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-01-16T04:41:53.869 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-01-16T04:41:53.869 INFO:teuthology.orchestra.run.smithi176.stderr:PUT 2024-01-16T04:41:53.869 INFO:teuthology.orchestra.run.smithi176.stderr:/test-client.0-q85zleljx4aymwl-211 2024-01-16T04:41:53.869 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:53.869 INFO:teuthology.orchestra.run.smithi176.stderr:host:smithi176.front.sepia.ceph.com 2024-01-16T04:41:53.869 INFO:teuthology.orchestra.run.smithi176.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:41:53.869 INFO:teuthology.orchestra.run.smithi176.stderr:x-amz-date:20240116T043849Z 2024-01-16T04:41:53.869 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:host;x-amz-content-sha256;x-amz-date 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: StringToSign: 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:AWS4-HMAC-SHA256 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:20240116T043849Z 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:20240116/us-east-1/s3/aws4_request 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:557f0eb9c11c30588bd14b9807b30306d194834f9f98ddca06b92486006c4961 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: Signature: 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:ef8871cec7e254f6e9a7ed84a20492b2605a764a2e9bc1bcbe7d24ebc1d2c5c7 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:urllib3.connectionpool: DEBUG: http://smithi176.front.sepia.ceph.com:80 "PUT /test-client.0-q85zleljx4aymwl-211 HTTP/1.1" 200 0 2024-01-16T04:41:53.870 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000c89d6b148b4fcfb1-0065a60859-1064-default', 'Content-Length': '0', 'Date': 'Tue, 16 Jan 2024 04:38:49 GMT'} 2024-01-16T04:41:53.871 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.parsers: DEBUG: Response body: 2024-01-16T04:41:53.871 INFO:teuthology.orchestra.run.smithi176.stderr:b'' 2024-01-16T04:41:53.871 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-01-16T04:41:53.871 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-01-16T04:41:53.871 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.871 INFO:teuthology.orchestra.run.smithi176.stderr:--------------------- >> end captured logging << --------------------- 2024-01-16T04:41:53.871 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:53.871 INFO:teuthology.orchestra.run.smithi176.stderr:====================================================================== 2024-01-16T04:41:53.871 INFO:teuthology.orchestra.run.smithi176.stderr:FAIL: s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl 2024-01-16T04:41:53.871 INFO:teuthology.orchestra.run.smithi176.stderr:---------------------------------------------------------------------- 2024-01-16T04:41:53.871 INFO:teuthology.orchestra.run.smithi176.stderr:Traceback (most recent call last): 2024-01-16T04:41:53.871 INFO:teuthology.orchestra.run.smithi176.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2024-01-16T04:41:53.872 INFO:teuthology.orchestra.run.smithi176.stderr: self.test(*self.arg) 2024-01-16T04:41:53.872 INFO:teuthology.orchestra.run.smithi176.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 4453, in test_bucket_recreate_new_acl 2024-01-16T04:41:53.872 INFO:teuthology.orchestra.run.smithi176.stderr: e = assert_raises(ClientError, client.create_bucket, Bucket=bucket_name, ACL='public-read') 2024-01-16T04:41:53.872 INFO:teuthology.orchestra.run.smithi176.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/utils.py", line 21, in assert_raises 2024-01-16T04:41:53.872 INFO:teuthology.orchestra.run.smithi176.stderr: raise AssertionError("%s not raised" % excName) 2024-01-16T04:41:53.872 INFO:teuthology.orchestra.run.smithi176.stderr:AssertionError: ClientError not raised 2024-01-16T04:41:53.872 INFO:teuthology.orchestra.run.smithi176.stderr:-------------------- >> begin captured logging << -------------------- 2024-01-16T04:41:53.872 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2024-01-16T04:41:53.872 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-01-16T04:41:53.872 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f7678ac4598> 2024-01-16T04:41:53.872 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-01-16T04:41:53.873 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2024-01-16T04:41:53.873 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2024-01-16T04:41:53.873 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:41:53.873 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.873 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.873 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:41:53.873 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:41:53.873 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.873 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:41:53.873 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:41:53.873 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-q85zleljx4aymwl-212', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-532.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi176.front.sepia.ceph.com:80/test-client.0-q85zleljx4aymwl-212', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-q85zleljx4aymwl-212'}}} 2024-01-16T04:41:53.873 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr:PUT 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr:/test-client.0-q85zleljx4aymwl-212 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr:host:smithi176.front.sepia.ceph.com 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr:x-amz-date:20240116T043849Z 2024-01-16T04:41:53.874 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:host;x-amz-content-sha256;x-amz-date 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: StringToSign: 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:AWS4-HMAC-SHA256 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:20240116T043849Z 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:20240116/us-east-1/s3/aws4_request 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:4069a349a7e4df5d49dc9fe49f386ff6890e75957625ef7a0f069bd9b739e414 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: Signature: 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:7aa652699253bebb6cede58b315b9b23b601e861e7088e1fbee0f69f76ebdc6f 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi176.front.sepia.ceph.com:80 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:urllib3.connectionpool: DEBUG: http://smithi176.front.sepia.ceph.com:80 "PUT /test-client.0-q85zleljx4aymwl-212 HTTP/1.1" 200 0 2024-01-16T04:41:53.875 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx0000008355ae82cf14921-0065a60859-1064-default', 'Content-Length': '0', 'Date': 'Tue, 16 Jan 2024 04:38:49 GMT'} 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.parsers: DEBUG: Response body: 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:b'' 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-q85zleljx4aymwl-212', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-acl': 'public-read', 'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-532.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi176.front.sepia.ceph.com:80/test-client.0-q85zleljx4aymwl-212', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-q85zleljx4aymwl-212'}}} 2024-01-16T04:41:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr:PUT 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr:/test-client.0-q85zleljx4aymwl-212 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr:host:smithi176.front.sepia.ceph.com 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr:x-amz-acl:public-read 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr:x-amz-date:20240116T043849Z 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:53.877 INFO:teuthology.orchestra.run.smithi176.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: StringToSign: 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:AWS4-HMAC-SHA256 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:20240116T043849Z 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:20240116/us-east-1/s3/aws4_request 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:ffd40edde2ecc25d0c535b0071cf030a67294f3dfbbebff7d756dda593565294 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.auth: DEBUG: Signature: 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:c6f39f2d997438c4ca74370f95e3aa9ac8c69e5460902a394203282d7c1a5d0f 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:urllib3.connectionpool: DEBUG: http://smithi176.front.sepia.ceph.com:80 "PUT /test-client.0-q85zleljx4aymwl-212 HTTP/1.1" 200 0 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx0000040b5909e1d7df0a4-0065a60859-1064-default', 'Content-Length': '0', 'Date': 'Tue, 16 Jan 2024 04:38:49 GMT'} 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.parsers: DEBUG: Response body: 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:b'' 2024-01-16T04:41:53.878 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-01-16T04:41:53.879 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-01-16T04:41:53.879 INFO:teuthology.orchestra.run.smithi176.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-01-16T04:41:53.879 INFO:teuthology.orchestra.run.smithi176.stderr:--------------------- >> end captured logging << --------------------- 2024-01-16T04:41:53.879 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:53.879 INFO:teuthology.orchestra.run.smithi176.stderr:---------------------------------------------------------------------- 2024-01-16T04:41:53.879 INFO:teuthology.orchestra.run.smithi176.stderr:Ran 566 tests in 224.693s 2024-01-16T04:41:53.879 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:53.879 INFO:teuthology.orchestra.run.smithi176.stderr:FAILED (SKIP=76, failures=2) 2024-01-16T04:41:53.936 INFO:teuthology.orchestra.run.smithi176.stderr:Exception ignored in: .generate at 0x7f767c464a40> 2024-01-16T04:41:53.936 INFO:teuthology.orchestra.run.smithi176.stderr:RuntimeError: generator ignored GeneratorExit 2024-01-16T04:41:53.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:41:53.970 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 441, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi176 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2024-01-16T04:41:53.971 INFO:tasks.s3tests:Cleaning up boto... 2024-01-16T04:41:53.972 DEBUG:teuthology.orchestra.run.smithi176:> rm /home/ubuntu/cephtest/boto.cfg 2024-01-16T04:41:53.990 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2024-01-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.079+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.080+0000 7f2f30dc27c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.080+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.080+0000 7f2f30dc27c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.080+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.080+0000 7f2f30dc27c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.080+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.080+0000 7f2f30dc27c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.088+0000 7f2f30dc27c0 20 RGWRados::pool_iterate: got zonegroup_info.ddb5a393-8a31-4788-b116-e494d90f23e0 2024-01-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.088+0000 7f2f30dc27c0 20 RGWRados::pool_iterate: got zone_info.9cd49eee-6ce1-47e3-8681-41594a956139 2024-01-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.088+0000 7f2f30dc27c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.088+0000 7f2f30dc27c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.088+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.088+0000 7f2f30dc27c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.088+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.088+0000 7f2f30dc27c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.088+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.088+0000 7f2f30dc27c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.088+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.089+0000 7f2f30dc27c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.089+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.089+0000 7f2f30dc27c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.089+0000 7f2f30dc27c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.089+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.089+0000 7f2f30dc27c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.089+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.089+0000 7f2f30dc27c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.089+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.089+0000 7f2f30dc27c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.089+0000 7f2f30dc27c0 20 zonegroup default 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.089+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.090+0000 7f2f30dc27c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.090+0000 7f2f30dc27c0 10 Cannot find current period zone using local zone 2024-01-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.090+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.090+0000 7f2f30dc27c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.090+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.090+0000 7f2f30dc27c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.090+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.090+0000 7f2f30dc27c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.090+0000 7f2f30dc27c0 20 zone default found 2024-01-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.090+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.090+0000 7f2f30dc27c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.090+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.091+0000 7f2f30dc27c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.091+0000 7f2f30dc27c0 20 started sync module instance, tier type = 2024-01-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.091+0000 7f2f30dc27c0 20 started zone id=9cd49eee-6ce1-47e3-8681-41594a956139 (name=default) with tier type = 2024-01-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.114+0000 7f2f30dc27c0 20 add_watcher() i=0 2024-01-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.114+0000 7f2f30dc27c0 20 add_watcher() i=1 2024-01-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.114+0000 7f2f30dc27c0 20 add_watcher() i=2 2024-01-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.114+0000 7f2f30dc27c0 20 add_watcher() i=3 2024-01-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.114+0000 7f2f30dc27c0 20 add_watcher() i=4 2024-01-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.114+0000 7f2f30dc27c0 20 add_watcher() i=5 2024-01-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.114+0000 7f2f30dc27c0 20 add_watcher() i=6 2024-01-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.115+0000 7f2f30dc27c0 20 add_watcher() i=7 2024-01-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.115+0000 7f2f30dc27c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.117+0000 7f2eed7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.117+0000 7f2f30dc27c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.117+0000 7f2f30dc27c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.171+0000 7f2e5bfff700 20 reqs_thread_entry: start 2024-01-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.171+0000 7f2e59ffb700 20 reqs_thread_entry: start 2024-01-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.171+0000 7f2f30dc27c0 20 init_complete bucket index max shards: 11 2024-01-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.171+0000 7f2f30dc27c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.171+0000 7f2e47fff700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:42:24 2024 2024-01-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.171+0000 7f2e477fe700 20 reqs_thread_entry: start 2024-01-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.171+0000 7f2f30dc27c0 20 get_system_obj_state: rctx=0x7ffcf0e6cad8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x562a55c61d50 s->prefetch_data=0 2024-01-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.171+0000 7f2f30dc27c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-01-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.172+0000 7f2f30dc27c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2024-01-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.172+0000 7f2f30dc27c0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:41:54.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.172+0000 7f2f30dc27c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:41:54.170 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.172+0000 7f2f30dc27c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:41:54.170 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.172+0000 7f2f30dc27c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:41:54.170 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.172+0000 7f2f30dc27c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.170 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.172+0000 7f2f30dc27c0 20 rados_obj.operate() r=0 bl.length=469 2024-01-16T04:41:54.170 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.172+0000 7f2f30dc27c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x11 2024-01-16T04:41:54.170 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.172+0000 7f2f30dc27c0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:41:54.170 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.173+0000 7f2f30dc27c0 10 removing key index: KRDTGHFZGBRBRVNHGFXA 2024-01-16T04:41:54.171 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.173+0000 7f2f30dc27c0 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.keys:KRDTGHFZGBRBRVNHGFXA, ofs0, ns] 2024-01-16T04:41:54.171 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.173+0000 7f2f19f1e700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824193 cookie 94739828992592 notifier 4238 bl.length()=170 2024-01-16T04:41:54.171 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.175+0000 7f2f30dc27c0 10 removing email index: foo.client.0+test@test.test 2024-01-16T04:41:54.171 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.175+0000 7f2f30dc27c0 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2024-01-16T04:41:54.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.175+0000 7f2f19f1e700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824194 cookie 94739828998048 notifier 4238 bl.length()=178 2024-01-16T04:41:54.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.178+0000 7f2f30dc27c0 10 removing user buckets index 2024-01-16T04:41:54.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.178+0000 7f2f30dc27c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2024-01-16T04:41:54.175 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.178+0000 7f2f1971d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824195 cookie 94739828984912 notifier 4238 bl.length()=169 2024-01-16T04:41:54.176 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.180+0000 7f2f30dc27c0 10 removing user index: foo.client.0 2024-01-16T04:41:54.177 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.180+0000 7f2f30dc27c0 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2024-01-16T04:41:54.177 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.180+0000 7f2f30dc27c0 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-01-16T04:41:54.177 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.180+0000 7f2f19f1e700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823212 cookie 94739828981312 notifier 4238 bl.length()=161 2024-01-16T04:41:54.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.183+0000 7f2f30dc27c0 20 remove_watcher() i=0 2024-01-16T04:41:54.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.183+0000 7f2f30dc27c0 2 removed watcher, disabling cache 2024-01-16T04:41:54.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.184+0000 7f2f30dc27c0 20 remove_watcher() i=1 2024-01-16T04:41:54.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.185+0000 7f2f30dc27c0 20 remove_watcher() i=2 2024-01-16T04:41:54.182 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.185+0000 7f2f30dc27c0 20 remove_watcher() i=3 2024-01-16T04:41:54.183 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.186+0000 7f2f30dc27c0 20 remove_watcher() i=4 2024-01-16T04:41:54.184 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.187+0000 7f2f30dc27c0 20 remove_watcher() i=5 2024-01-16T04:41:54.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.188+0000 7f2f30dc27c0 20 remove_watcher() i=6 2024-01-16T04:41:54.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.189+0000 7f2f30dc27c0 20 remove_watcher() i=7 2024-01-16T04:41:54.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.189+0000 7f2eed7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:41:54.192 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2024-01-16T04:41:54.273 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:41:54.273 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:41:54.279 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.282+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.280 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.283+0000 7f34e06987c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.280 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.283+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.280 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.283+0000 7f34e06987c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.280 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.283+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.280 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.283+0000 7f34e06987c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.280 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.283+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.281 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.284+0000 7f34e06987c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.290+0000 7f34e06987c0 20 RGWRados::pool_iterate: got zonegroup_info.ddb5a393-8a31-4788-b116-e494d90f23e0 2024-01-16T04:41:54.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.290+0000 7f34e06987c0 20 RGWRados::pool_iterate: got zone_info.9cd49eee-6ce1-47e3-8681-41594a956139 2024-01-16T04:41:54.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.290+0000 7f34e06987c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:41:54.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.290+0000 7f34e06987c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:41:54.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.290+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.291+0000 7f34e06987c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.291+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.291+0000 7f34e06987c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:41:54.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.291+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.291+0000 7f34e06987c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.291+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.290 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.291+0000 7f34e06987c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:41:54.290 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.291+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.290 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.290 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 20 zonegroup default 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 10 Cannot find current period zone using local zone 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.292+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.293+0000 7f34e06987c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.293+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.293+0000 7f34e06987c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.293+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.293+0000 7f34e06987c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:41:54.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.293+0000 7f34e06987c0 20 zone default found 2024-01-16T04:41:54.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.293+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.293+0000 7f34e06987c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.293+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.293+0000 7f34e06987c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.293+0000 7f34e06987c0 20 started sync module instance, tier type = 2024-01-16T04:41:54.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.293+0000 7f34e06987c0 20 started zone id=9cd49eee-6ce1-47e3-8681-41594a956139 (name=default) with tier type = 2024-01-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.320+0000 7f34e06987c0 20 add_watcher() i=0 2024-01-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.320+0000 7f34e06987c0 20 add_watcher() i=1 2024-01-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.320+0000 7f34e06987c0 20 add_watcher() i=2 2024-01-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.320+0000 7f34e06987c0 20 add_watcher() i=3 2024-01-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.320+0000 7f34e06987c0 20 add_watcher() i=4 2024-01-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.320+0000 7f34e06987c0 20 add_watcher() i=5 2024-01-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.320+0000 7f34e06987c0 20 add_watcher() i=6 2024-01-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.321+0000 7f34e06987c0 20 add_watcher() i=7 2024-01-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.321+0000 7f34e06987c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.322+0000 7f3417fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.322+0000 7f34e06987c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.322+0000 7f34e06987c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:41:54.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f3406ffd700 20 reqs_thread_entry: start 2024-01-16T04:41:54.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f3404ff9700 20 reqs_thread_entry: start 2024-01-16T04:41:54.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f34e06987c0 20 init_complete bucket index max shards: 11 2024-01-16T04:41:54.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f34e06987c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:41:54.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f33feffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:42:24 2024 2024-01-16T04:41:54.362 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:54.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f33fe7fc700 20 reqs_thread_entry: start 2024-01-16T04:41:54.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f34e06987c0 20 get_system_obj_state: rctx=0x7ffcd0424138 obj=default.rgw.meta:users.uid:bar.client.0 state=0x561408a85d50 s->prefetch_data=0 2024-01-16T04:41:54.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f34e06987c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-01-16T04:41:54.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f34e06987c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2024-01-16T04:41:54.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f34e06987c0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:41:54.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f34e06987c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:41:54.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f34e06987c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:41:54.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f34e06987c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:41:54.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.364+0000 7f34e06987c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.365+0000 7f34e06987c0 20 rados_obj.operate() r=0 bl.length=469 2024-01-16T04:41:54.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.365+0000 7f34e06987c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x11 2024-01-16T04:41:54.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.365+0000 7f34e06987c0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:41:54.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.365+0000 7f34e06987c0 10 removing key index: KIFEZSRMUIJENLUODLLJ 2024-01-16T04:41:54.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.365+0000 7f34e06987c0 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.keys:KIFEZSRMUIJENLUODLLJ, ofs0, ns] 2024-01-16T04:41:54.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.365+0000 7f34c97f4700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824196 cookie 94644045910880 notifier 4244 bl.length()=170 2024-01-16T04:41:54.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.367+0000 7f34e06987c0 10 removing email index: bar.client.0+test@test.test 2024-01-16T04:41:54.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.367+0000 7f34e06987c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2024-01-16T04:41:54.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.367+0000 7f34c97f4700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824197 cookie 94644045903120 notifier 4244 bl.length()=178 2024-01-16T04:41:54.365 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.368+0000 7f34e06987c0 10 removing user buckets index 2024-01-16T04:41:54.365 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.368+0000 7f34e06987c0 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2024-01-16T04:41:54.365 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.368+0000 7f34c97f4700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823703 cookie 94644045907728 notifier 4244 bl.length()=169 2024-01-16T04:41:54.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.371+0000 7f34e06987c0 10 removing user index: bar.client.0 2024-01-16T04:41:54.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.371+0000 7f34e06987c0 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2024-01-16T04:41:54.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.371+0000 7f34e06987c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-01-16T04:41:54.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.371+0000 7f34c97f4700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824198 cookie 94644045903120 notifier 4244 bl.length()=161 2024-01-16T04:41:54.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.374+0000 7f34e06987c0 20 remove_watcher() i=0 2024-01-16T04:41:54.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.374+0000 7f34e06987c0 2 removed watcher, disabling cache 2024-01-16T04:41:54.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.375+0000 7f34e06987c0 20 remove_watcher() i=1 2024-01-16T04:41:54.374 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.377+0000 7f34e06987c0 20 remove_watcher() i=2 2024-01-16T04:41:54.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.380+0000 7f34e06987c0 20 remove_watcher() i=3 2024-01-16T04:41:54.377 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.381+0000 7f34e06987c0 20 remove_watcher() i=4 2024-01-16T04:41:54.378 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.382+0000 7f34e06987c0 20 remove_watcher() i=5 2024-01-16T04:41:54.380 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.384+0000 7f34e06987c0 20 remove_watcher() i=6 2024-01-16T04:41:54.383 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.387+0000 7f34e06987c0 20 remove_watcher() i=7 2024-01-16T04:41:54.383 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.387+0000 7f3417fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:41:54.388 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2024-01-16T04:41:54.470 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:41:54.470 INFO:teuthology.orchestra.run.smithi176.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:41:54.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.479+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.480+0000 7fcc91e9e7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.480+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.480+0000 7fcc91e9e7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.480+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.480+0000 7fcc91e9e7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.480+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.480+0000 7fcc91e9e7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.487+0000 7fcc91e9e7c0 20 RGWRados::pool_iterate: got zonegroup_info.ddb5a393-8a31-4788-b116-e494d90f23e0 2024-01-16T04:41:54.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.487+0000 7fcc91e9e7c0 20 RGWRados::pool_iterate: got zone_info.9cd49eee-6ce1-47e3-8681-41594a956139 2024-01-16T04:41:54.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.487+0000 7fcc91e9e7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:41:54.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.487+0000 7fcc91e9e7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:41:54.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.487+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.487+0000 7fcc91e9e7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.487+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.487+0000 7fcc91e9e7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.487+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.488+0000 7fcc91e9e7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.488+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.488+0000 7fcc91e9e7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.488+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.488+0000 7fcc91e9e7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.488+0000 7fcc91e9e7c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.488+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.488+0000 7fcc91e9e7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.488+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.488+0000 7fcc91e9e7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.488+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:41:54.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 20 zonegroup default 2024-01-16T04:41:54.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 10 Cannot find current period zone using local zone 2024-01-16T04:41:54.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:41:54.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:41:54.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 20 zone default found 2024-01-16T04:41:54.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.489+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.490+0000 7fcc91e9e7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.490+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.490+0000 7fcc91e9e7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:41:54.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.490+0000 7fcc91e9e7c0 20 started sync module instance, tier type = 2024-01-16T04:41:54.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.490+0000 7fcc91e9e7c0 20 started zone id=9cd49eee-6ce1-47e3-8681-41594a956139 (name=default) with tier type = 2024-01-16T04:41:54.509 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.512+0000 7fcc91e9e7c0 20 add_watcher() i=0 2024-01-16T04:41:54.509 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.512+0000 7fcc91e9e7c0 20 add_watcher() i=1 2024-01-16T04:41:54.509 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.512+0000 7fcc91e9e7c0 20 add_watcher() i=2 2024-01-16T04:41:54.509 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.512+0000 7fcc91e9e7c0 20 add_watcher() i=3 2024-01-16T04:41:54.509 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.512+0000 7fcc91e9e7c0 20 add_watcher() i=4 2024-01-16T04:41:54.509 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.512+0000 7fcc91e9e7c0 20 add_watcher() i=5 2024-01-16T04:41:54.509 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.512+0000 7fcc91e9e7c0 20 add_watcher() i=6 2024-01-16T04:41:54.511 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.515+0000 7fcc91e9e7c0 20 add_watcher() i=7 2024-01-16T04:41:54.512 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.515+0000 7fcc91e9e7c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:41:54.515 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.518+0000 7fcc4dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:41:54.515 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.518+0000 7fcc91e9e7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:41:54.515 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.518+0000 7fcc91e9e7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:41:54.581 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.583+0000 7fcbc4ff9700 20 reqs_thread_entry: start 2024-01-16T04:41:54.581 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.583+0000 7fcbba7fc700 20 reqs_thread_entry: start 2024-01-16T04:41:54.581 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.583+0000 7fcc91e9e7c0 20 init_complete bucket index max shards: 11 2024-01-16T04:41:54.581 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.583+0000 7fcc91e9e7c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:41:54.581 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.583+0000 7fcbb8ff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:42:24 2024 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.583+0000 7fcb9ffff700 20 reqs_thread_entry: start 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.584+0000 7fcc91e9e7c0 20 get_system_obj_state: rctx=0x7ffed0498538 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55933943cd50 s->prefetch_data=0 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.584+0000 7fcc91e9e7c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.584+0000 7fcc91e9e7c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.584+0000 7fcc91e9e7c0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.584+0000 7fcc91e9e7c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.584+0000 7fcc91e9e7c0 10 updating xattr: name=user.rgw.user-policy bl.length()=118 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.584+0000 7fcc91e9e7c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.584+0000 7fcc91e9e7c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.584+0000 7fcc91e9e7c0 20 rados->read ofs=0 len=0 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.584+0000 7fcc91e9e7c0 20 rados_obj.operate() r=0 bl.length=528 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.584+0000 7fcc91e9e7c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x11 2024-01-16T04:41:54.582 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.584+0000 7fcc91e9e7c0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:41:54.583 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.585+0000 7fcc91e9e7c0 10 removing key index: PLSGDJEUQHITTOSCZDQL 2024-01-16T04:41:54.583 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.585+0000 7fcc91e9e7c0 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:PLSGDJEUQHITTOSCZDQL, ofs0, ns] 2024-01-16T04:41:54.583 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.585+0000 7fcc767fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823704 cookie 94090810624144 notifier 4250 bl.length()=170 2024-01-16T04:41:54.583 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.586+0000 7fcc91e9e7c0 10 removing email index: testx$tenanteduser.client.0+test@test.test 2024-01-16T04:41:54.584 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.586+0000 7fcc91e9e7c0 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2024-01-16T04:41:54.584 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.586+0000 7fcc75ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823213 cookie 94090810636832 notifier 4250 bl.length()=193 2024-01-16T04:41:54.584 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.588+0000 7fcc91e9e7c0 10 removing user buckets index 2024-01-16T04:41:54.584 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.588+0000 7fcc91e9e7c0 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2024-01-16T04:41:54.584 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.588+0000 7fcc767fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824199 cookie 94090810617776 notifier 4250 bl.length()=184 2024-01-16T04:41:54.586 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.589+0000 7fcc91e9e7c0 10 removing user index: testx$tenanteduser.client.0 2024-01-16T04:41:54.586 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.589+0000 7fcc91e9e7c0 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2024-01-16T04:41:54.586 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.589+0000 7fcc91e9e7c0 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-01-16T04:41:54.586 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.589+0000 7fcc75ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823214 cookie 94090810632000 notifier 4250 bl.length()=176 2024-01-16T04:41:54.589 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.592+0000 7fcc91e9e7c0 20 remove_watcher() i=0 2024-01-16T04:41:54.589 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.592+0000 7fcc91e9e7c0 2 removed watcher, disabling cache 2024-01-16T04:41:54.590 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.593+0000 7fcc91e9e7c0 20 remove_watcher() i=1 2024-01-16T04:41:54.590 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.594+0000 7fcc91e9e7c0 20 remove_watcher() i=2 2024-01-16T04:41:54.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.595+0000 7fcc91e9e7c0 20 remove_watcher() i=3 2024-01-16T04:41:54.592 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.596+0000 7fcc91e9e7c0 20 remove_watcher() i=4 2024-01-16T04:41:54.593 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.597+0000 7fcc91e9e7c0 20 remove_watcher() i=5 2024-01-16T04:41:54.594 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.597+0000 7fcc91e9e7c0 20 remove_watcher() i=6 2024-01-16T04:41:54.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.598+0000 7fcc91e9e7c0 20 remove_watcher() i=7 2024-01-16T04:41:54.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-01-16T04:41:54.599+0000 7fcc4dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:41:54.600 INFO:tasks.s3tests:Removing s3-tests... 2024-01-16T04:41:54.600 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/s3-tests 2024-01-16T04:41:54.756 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 639, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 59, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 441, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi176 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2024-01-16T04:41:54.986 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c81f3286ef6d469dbcee02d8ed992fa0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 639, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 59, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 441, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi176 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2024-01-16T04:41:54.989 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2024-01-16T04:41:55.000 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-01-16T04:41:55.099 DEBUG:tasks.rgw.client.0:waiting for process to exit 2024-01-16T04:41:55.099 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:41:55.104 INFO:tasks.rgw.client.0.smithi176.stdout:2024-01-16T04:41:55.103+0000 7f3c381ae700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends civetweb port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi176.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --foreground (PID: 43229) UID: 0 2024-01-16T04:41:55.104 INFO:tasks.rgw.client.0.smithi176.stdout:2024-01-16T04:41:55.103+0000 7f3c4feaa880 -1 shutting down 2024-01-16T04:41:59.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2024-01-16T04:42:01.200 INFO:tasks.rgw.client.0:Stopped 2024-01-16T04:42:01.200 DEBUG:teuthology.orchestra.run.smithi176:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2024-01-16T04:42:01.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/vault-root-token 2024-01-16T04:42:01.297 DEBUG:teuthology.run_tasks:Unwinding manager vault 2024-01-16T04:42:01.306 INFO:tasks.vault:Stopping Vault instance 2024-01-16T04:42:01.306 DEBUG:tasks.vault.client.0:waiting for process to exit 2024-01-16T04:42:01.306 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:42:01.339 INFO:tasks.vault.client.0.smithi176.stdout:==> Vault shutdown triggered 2024-01-16T04:42:01.340 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:42:01.343Z [INFO] core: marked as sealed 2024-01-16T04:42:01.340 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:42:01.343Z [INFO] core: pre-seal teardown starting 2024-01-16T04:42:01.340 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:42:01.343Z [INFO] rollback: stopping rollback manager 2024-01-16T04:42:01.340 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:42:01.343Z [INFO] core: pre-seal teardown complete 2024-01-16T04:42:01.340 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:42:01.343Z [INFO] core: stopping cluster listeners 2024-01-16T04:42:01.341 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:42:01.343Z [INFO] core.cluster-listener: forwarding rpc listeners stopped 2024-01-16T04:42:01.541 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:42:01.546Z [INFO] core.cluster-listener: rpc listeners successfully shut down 2024-01-16T04:42:01.542 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:42:01.546Z [INFO] core: cluster listeners successfully shut down 2024-01-16T04:42:01.542 INFO:tasks.vault.client.0.smithi176.stderr:2024-01-16T04:42:01.546Z [INFO] core: vault is sealed 2024-01-16T04:42:01.543 INFO:tasks.vault.client.0:Stopped 2024-01-16T04:42:01.543 INFO:tasks.vault:Removing Vault... 2024-01-16T04:42:01.543 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/vault /home/ubuntu/cephtest/vault.zip 2024-01-16T04:42:01.620 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-16T04:42:01.630 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-01-16T04:42:01.630 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T04:42:01.954 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-16T04:42:01.955 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":36,"fsid":"43305b7c-7b81-41aa-8897-fc8a960bca4f","created":"2024-01-16T04:36:40.444462+0000","modified":"2024-01-16T04:38:47.422294+0000","last_up_change":"2024-01-16T04:36:46.843164+0000","last_in_change":"2024-01-16T04:36:41.376078+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-16T04:36:46.881958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-16T04:36:49.062116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2024-01-16T04:37:09.282117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2024-01-16T04:37:10.774698+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2024-01-16T04:37:12.911770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2024-01-16T04:37:14.525058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2024-01-16T04:37:16.497960+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2024-01-16T04:37:18.564482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2024-01-16T04:37:29.424301+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2024-01-16T04:38:10.373321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"847d61e2-90f6-4b18-9185-770c7d2aa786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":40345},{"type":"v1","addr":"172.21.15.176:6801","nonce":40345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":40345},{"type":"v1","addr":"172.21.15.176:6803","nonce":40345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":40345},{"type":"v1","addr":"172.21.15.176:6807","nonce":40345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":40345},{"type":"v1","addr":"172.21.15.176:6805","nonce":40345}]},"public_addr":"172.21.15.176:6801/40345","cluster_addr":"172.21.15.176:6803/40345","heartbeat_back_addr":"172.21.15.176:6807/40345","heartbeat_front_addr":"172.21.15.176:6805/40345","state":["exists","up"]},{"osd":1,"uuid":"d5421cb7-0f31-4225-8ae7-ab6b68e16c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":40360},{"type":"v1","addr":"172.21.15.176:6810","nonce":40360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":40360},{"type":"v1","addr":"172.21.15.176:6814","nonce":40360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":40360},{"type":"v1","addr":"172.21.15.176:6822","nonce":40360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":40360},{"type":"v1","addr":"172.21.15.176:6818","nonce":40360}]},"public_addr":"172.21.15.176:6810/40360","cluster_addr":"172.21.15.176:6814/40360","heartbeat_back_addr":"172.21.15.176:6822/40360","heartbeat_front_addr":"172.21.15.176:6818/40360","state":["exists","up"]},{"osd":2,"uuid":"9e5cf429-dd1d-41ee-a3d0-36013414c1fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":40361},{"type":"v1","addr":"172.21.15.176:6811","nonce":40361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":40361},{"type":"v1","addr":"172.21.15.176:6815","nonce":40361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":40361},{"type":"v1","addr":"172.21.15.176:6823","nonce":40361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":40361},{"type":"v1","addr":"172.21.15.176:6819","nonce":40361}]},"public_addr":"172.21.15.176:6811/40361","cluster_addr":"172.21.15.176:6815/40361","heartbeat_back_addr":"172.21.15.176:6823/40361","heartbeat_front_addr":"172.21.15.176:6819/40361","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:36:45.817427+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:36:45.844338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:36:45.794505+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":1}]},{"pgid":"3.3c","mappings":[{"from":0,"to":1}]},{"pgid":"4.4","mappings":[{"from":1,"to":0}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.15","mappings":[{"from":1,"to":2}]},{"pgid":"4.25","mappings":[{"from":1,"to":2}]},{"pgid":"4.29","mappings":[{"from":1,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":2}]},{"pgid":"10.1b","mappings":[{"from":2,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:42:01.968 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-16T04:42:02.267 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-16T04:42:02.267 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-16T04:42:02.267 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-16T04:42:02.267 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:42:02.313 INFO:tasks.ceph.osd.0:Stopped 2024-01-16T04:42:02.313 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-16T04:42:02.313 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:42:02.356 INFO:tasks.ceph.osd.1:Stopped 2024-01-16T04:42:02.357 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-16T04:42:02.357 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:42:02.401 INFO:tasks.ceph.osd.2:Stopped 2024-01-16T04:42:02.401 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-16T04:42:02.401 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-16T04:42:02.401 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:42:02.422 INFO:tasks.ceph.mgr.x:Stopped 2024-01-16T04:42:02.422 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-16T04:42:02.422 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-16T04:42:02.422 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:42:02.472 INFO:tasks.ceph.mon.a:Stopped 2024-01-16T04:42:02.472 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-16T04:42:02.472 DEBUG:teuthology.orchestra.run.smithi176:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-01-16T04:42:02.541 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi176.front.sepia.ceph.com 2024-01-16T04:42:02.541 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-16T04:42:04.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2024-01-16T04:42:04.933 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi176.front.sepia.ceph.com 2024-01-16T04:42:04.933 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-16T04:42:05.033 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi176.front.sepia.ceph.com 2024-01-16T04:42:05.033 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-16T04:42:05.112 INFO:tasks.ceph:Archiving mon data... 2024-01-16T04:42:05.114 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517322/data/mon.a.tgz 2024-01-16T04:42:05.114 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-01-16T04:42:05.132 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.TvrDyrH37S 2024-01-16T04:42:05.133 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.TvrDyrH37S 2024-01-16T04:42:05.388 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.TvrDyrH37S 2024-01-16T04:42:05.475 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.TvrDyrH37S is 1MB 2024-01-16T04:42:05.585 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.TvrDyrH37S 2024-01-16T04:42:05.603 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-16T04:42:05.604 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-16T04:42:05.669 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-01-16T04:42:05.670 DEBUG:teuthology.orchestra.run.smithi176:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-01-16T04:42:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-01-16T04:42:05.742 INFO:tasks.ceph:Archiving crash dumps... 2024-01-16T04:42:05.744 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517322/remote/smithi176/crash 2024-01-16T04:42:05.745 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-16T04:42:05.808 INFO:tasks.ceph:Compressing logs... 2024-01-16T04:42:05.808 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T04:42:05.884 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.36413.log 2024-01-16T04:42:05.885 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42419.log 2024-01-16T04:42:05.885 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.tmp-client.admin.36413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43009.log 2024-01-16T04:42:05.885 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.36413.log.gz 2024-01-16T04:42:05.885 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42419.log.gz 2024-01-16T04:42:05.885 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42664.log 2024-01-16T04:42:05.886 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.43009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42468.log 2024-01-16T04:42:05.886 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43009.log.gz 2024-01-16T04:42:05.886 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42664.log.gz 2024-01-16T04:42:05.886 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/rgw.ceph.client.0.log 2024-01-16T04:42:05.886 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42468.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-16T04:42:05.886 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42468.log.gz 2024-01-16T04:42:05.886 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/rgw.ceph.client.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44472.log 2024-01-16T04:42:05.887 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42019.log 2024-01-16T04:42:05.887 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.0.44472.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-16T04:42:05.887 INFO:teuthology.orchestra.run.smithi176.stderr: 83.6% -- replaced with /var/log/ceph/ceph-client.0.44472.log.gz 2024-01-16T04:42:05.887 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42019.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.0.44178.log 2024-01-16T04:42:05.887 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42019.log.gz 2024-01-16T04:42:05.887 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43058.log 2024-01-16T04:42:05.887 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.0.44178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46525.log 2024-01-16T04:42:05.887 INFO:teuthology.orchestra.run.smithi176.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.0.44178.log.gz 2024-01-16T04:42:05.888 INFO:teuthology.orchestra.run.smithi176.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-16T04:42:05.888 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.43058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39936.log 2024-01-16T04:42:05.888 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43058.log.gz 2024-01-16T04:42:05.888 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.0.46525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42962.log 2024-01-16T04:42:05.888 INFO:teuthology.orchestra.run.smithi176.stderr: 83.4% -- replaced with /var/log/ceph/ceph-client.0.46525.log.gz 2024-01-16T04:42:05.888 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41901.log 2024-01-16T04:42:05.888 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39936.log.gz 2024-01-16T04:42:05.889 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42962.log.gz 2024-01-16T04:42:05.889 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40171.log 2024-01-16T04:42:05.889 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41901.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-16T04:42:05.889 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41901.log.gz 2024-01-16T04:42:05.889 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40171.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40171.log.gz-- 2024-01-16T04:42:05.889 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.0.46623.log 2024-01-16T04:42:05.889 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42068.log 2024-01-16T04:42:05.890 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.0.46623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42174.log 2024-01-16T04:42:05.890 INFO:teuthology.orchestra.run.smithi176.stderr: 83.4% -- replaced with /var/log/ceph/ceph-client.0.46623.log.gz 2024-01-16T04:42:05.890 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40122.log 2024-01-16T04:42:05.890 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42068.log.gz 2024-01-16T04:42:05.890 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42174.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-16T04:42:05.890 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42174.log.gz 2024-01-16T04:42:05.890 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40122.log.gz 2024-01-16T04:42:05.890 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42517.log 2024-01-16T04:42:05.891 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42223.log 2024-01-16T04:42:05.891 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46836.log 2024-01-16T04:42:05.891 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42517.log.gz 2024-01-16T04:42:05.891 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42223.log.gz 2024-01-16T04:42:05.891 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42370.log 2024-01-16T04:42:05.891 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42913.log 2024-01-16T04:42:05.891 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.46836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46836.log.gz 2024-01-16T04:42:05.892 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44276.log 2024-01-16T04:42:05.892 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42370.log.gz 2024-01-16T04:42:05.892 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42913.log.gz 2024-01-16T04:42:05.892 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.0.44276.log: 83.4% -- replaced with /var/log/ceph/ceph-client.0.44276.log.gz 2024-01-16T04:42:05.892 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42615.log 2024-01-16T04:42:05.892 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-16T04:42:05.892 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42615.log.gz 2024-01-16T04:42:05.897 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42566.log 2024-01-16T04:42:05.897 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42713.log 2024-01-16T04:42:05.897 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42566.log.gz 2024-01-16T04:42:05.898 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42713.log.gz 2024-01-16T04:42:05.898 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46883.log 2024-01-16T04:42:05.898 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2024-01-16T04:42:05.898 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.46883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46883.log.gz 2024-01-16T04:42:05.898 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2024-01-16T04:42:05.898 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-16T04:42:05.898 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42117.log 2024-01-16T04:42:05.899 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.42117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42117.log.gz 2024-01-16T04:42:05.899 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44374.log 2024-01-16T04:42:05.899 INFO:teuthology.orchestra.run.smithi176.stderr: 86.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-16T04:42:05.899 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.0.44374.log: 86.2% -- replaced with /var/log/ceph/ceph-client.0.44374.log.gz 2024-01-16T04:42:05.899 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40218.log 2024-01-16T04:42:05.899 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46427.log 2024-01-16T04:42:05.899 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40218.log.gz 2024-01-16T04:42:05.900 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.0.46427.log: 83.4% -- replaced with /var/log/ceph/ceph-client.0.46427.log.gz 2024-01-16T04:42:05.900 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44078.log 2024-01-16T04:42:05.900 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43980.log 2024-01-16T04:42:05.900 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.0.44078.log: 83.1% -- replaced with /var/log/ceph/ceph-client.0.44078.log.gz 2024-01-16T04:42:05.900 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.0.43980.log: 86.1% -- replaced with /var/log/ceph/ceph-client.0.43980.log.gz 2024-01-16T04:42:05.900 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42321.log 2024-01-16T04:42:05.900 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40344.log 2024-01-16T04:42:05.901 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42321.log.gz 2024-01-16T04:42:05.901 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40344.log.gz 2024-01-16T04:42:05.901 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-16T04:42:05.901 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42272.log 2024-01-16T04:42:05.901 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.42272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42272.log.gz 2024-01-16T04:42:05.901 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41970.log 2024-01-16T04:42:05.901 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41970.log.gz 2024-01-16T04:42:05.914 INFO:teuthology.orchestra.run.smithi176.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-16T04:42:06.003 INFO:teuthology.orchestra.run.smithi176.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-16T04:42:07.064 INFO:teuthology.orchestra.run.smithi176.stderr: 93.9% -- replaced with /var/log/ceph/rgw.ceph.client.0.log.gz 2024-01-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi176.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-16T04:42:12.129 INFO:teuthology.orchestra.run.smithi176.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-16T04:42:13.158 INFO:teuthology.orchestra.run.smithi176.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-16T04:42:13.160 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-01-16T04:42:13.160 INFO:teuthology.orchestra.run.smithi176.stderr:real 0m7.294s 2024-01-16T04:42:13.160 INFO:teuthology.orchestra.run.smithi176.stderr:user 0m18.716s 2024-01-16T04:42:13.160 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m0.742s 2024-01-16T04:42:13.160 INFO:tasks.ceph:Archiving logs... 2024-01-16T04:42:13.161 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517322/remote/smithi176/log 2024-01-16T04:42:13.162 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T04:42:15.951 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-16T04:42:15.961 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-16T04:42:15.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T04:42:16.023 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-16T04:42:16.042 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-16T04:42:16.043 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T04:42:16.065 INFO:teuthology.orchestra.run.smithi176.stderr:bash: ntpq: command not found 2024-01-16T04:42:16.072 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T04:42:16.073 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-01-16T04:42:16.073 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +2492us[+2492us] +/- 83ms 2024-01-16T04:42:16.073 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 +69us[ +68us] +/- 67ms 2024-01-16T04:42:16.074 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -2625us[-2625us] +/- 84ms 2024-01-16T04:42:16.074 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:42:16.075 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-16T04:42:16.101 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-16T04:42:16.102 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-16T04:42:16.147 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T04:42:16.211 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T04:42:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:42:16.242 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-16T04:42:16.252 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-16T04:42:16.267 INFO:teuthology.task.internal:Duration was 903.650086 seconds 2024-01-16T04:42:16.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-16T04:42:16.286 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-16T04:42:16.287 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T04:42:16.335 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T04:42:16.605 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-16T04:42:16.605 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-01-16T04:42:16.605 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T04:42:16.626 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-16T04:42:16.627 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T04:42:16.707 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-16T04:42:16.708 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T04:42:16.814 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-16T04:42:16.825 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-16T04:42:16.825 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T04:42:16.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-16T04:42:16.862 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T04:42:16.917 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-01-16T04:42:16.967 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T04:42:17.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:42:17.023 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-16T04:42:17.033 INFO:teuthology.task.internal:Transferring archived files... 2024-01-16T04:42:17.034 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517322/remote/smithi176 2024-01-16T04:42:17.034 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T04:42:17.102 INFO:teuthology.task.internal:Removing archive directory... 2024-01-16T04:42:17.103 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T04:42:17.152 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-16T04:42:17.164 INFO:teuthology.task.internal:Not uploading archives. 2024-01-16T04:42:17.164 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-16T04:42:17.182 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-16T04:42:17.182 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T04:42:17.209 INFO:teuthology.orchestra.run.smithi176.stdout: 525947 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 04:42 /home/ubuntu/cephtest 2024-01-16T04:42:17.211 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-16T04:42:17.221 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-16T04:42:17.242 INFO:teuthology.nuke:Checking targets against current locks 2024-01-16T04:42:17.264 DEBUG:teuthology.nuke:shortname: smithi176 2024-01-16T04:42:17.265 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T04:42:17.287 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 04:18:49.710715', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMIJErr9dGrhWCSEySf7+DUoB2WqQZ+JCMm6HItnXIkSTf0om2vY9E3f8+9olOrKYtE9tuLk7Llog4RR4LbdBc='} 2024-01-16T04:42:17.338 INFO:teuthology.orchestra.console.smithi176:Power off 2024-01-16T04:42:17.339 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T04:42:17.383 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-01-16T04:42:17.384 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T04:42:17.410 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-01-16T04:42:21.411 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T04:42:21.521 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-01-16T04:42:25.523 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T04:42:25.633 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-01-16T04:42:29.634 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T04:42:29.744 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-01-16T04:42:29.845 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-01-16T04:42:29.951 INFO:teuthology.run:Summary data: description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} duration: 903.6500861644745 failure_reason: 'Command failed (s3 tests against rgw) on smithi176 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a ''!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain''"' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c81f3286ef6d469dbcee02d8ed992fa0 status: fail success: false 2024-01-16T04:42:29.952 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T04:42:30.035 INFO:teuthology.run:FAIL